Automated evaluation and automated video production of combat sports events

Abstract

A computer-implemented method is provided for the automated detection of motion and physical contact between participants in sporting events. An image of the playing field is captured by a video camera system, which generates at least one video stream. This video stream is digitally processed to identify human movements and any associated contact between participants. Motion and contact detection is performed using an artificial neural network method without intermediate stages such as keypoint and human pose estimation.

Description

Technical Field

[0001] This disclosure generally relates to the field of video processing technology, and more specifically, to the evaluation of combat sports events such as boxing, mixed martial arts (MMA), wrestling, kickboxing, taekwondo, and Brazilian Jiu-Jitsu (BJJ), and the automation of video production for such combat sports events. The invention is designed for the automated and / or computer-aided production and / or evaluation of video streams captured within combat sports arenas (such as rings, cages, mats, etc.). The invention is also applicable to purposes including, but not limited to, referee assistance systems, match analysis, and statistical information. Background Technology

[0002] Traditionally, quantifying the combat activities performed by fighters in combat sports (such as punches, kicks, takedowns, etc.) requires a significant amount of time-consuming and post-fight manual work. Post-fight quantification of combat activities is typically done using performance analysts who manually go through every frame of the video recording of a combat sports event step-by-step to identify and label all relevant combat activities. This is a very accurate method, but also very expensive and impossible to perform in real time, as combat sports events can contain thousands of combat activities, each of which may require careful slow-motion review to be properly categorized.

[0003] To quantify combat sports events in real time, the industry has begun to rely on human real-time clickers, typically two to five per event, depending on the type of combat sport. Instead of reviewing the fight in slow motion afterward, human real-time clickers watch the event in real time like everyone else, but use clicker equipment to capture the most critical combat actions, such as swings and / or strikes. Even short combat sports events often involve hundreds of swings, with a 12-round boxing match typically including over 1,000 swings. During exchanges, there can sometimes be 20 or more strikes within seconds. Due to the fast-paced nature of combat sports, the human clicker method for quantifying combat actions in real time is generally considered highly inaccurate due to human limitations in reaction time and focus. Furthermore, due to constraints on available manpower and budget, typically only a few of the most relevant metrics are measured by the clickers. Therefore, human real-time clickers are primarily used as entertainment tools to enrich the audience experience when broadcasting large-scale combat sports events, and their use as performance tools for coaches and fighters, or as tools for judging combat sports events, has been found to be very limited due to low accuracy and high cost of use.

[0004] Recently, new methods have been developed for generating statistics on combat sports events in a more accurate and affordable manner. One such method is to add hardware sensors to athletes' equipment, such as their boxing gloves, see, for example, CN 116665104 A1, CN 116688473 A1, US 10124210 B2 and US 8485879 B2.

[0005] Another recently explored approach is to use computer vision and deep learning methods to identify striking events in video sources, see US 11514677 B2. The approach taken in US 11514677 B2 is to use keypoint and human pose estimators for person-to-person contact detection and activity detection. The advantage of using keypoint and human pose estimators is that one can use readily available models pre-trained on large public datasets and then fine-tune the keypoint or pose estimator for the corresponding local application. Using this method, one achieves a dimensionality-reduced feature space on which a classifier can be trained to obtain reasonable results even with very limited training data.

[0006] However, in the context of combat sports and contact detection, keypoint and human pose estimation-based methods (whether 2D or 3D) struggle to achieve good accuracy, especially when only a single video source is provided as input. This is primarily due to occlusion, self-occlusion, and feature filtering. When reducing the dimensionality of the input feature space of the source frame by creating intermediate representations in the form of keypoints or pose estimations, much information present in the original input frame is also filtered out, which can be crucial for obtaining the correct classification corresponding to what performance analysts would select. Therefore, while keypoint and human pose estimation-based methods may be advantageous when limited training data is required, they may lack predictive accuracy and robustness, particularly when applied to person-to-person contact detection in combat sports.

[0007] Therefore, the first technical problem on which this invention is based is to provide a method for automated evaluation of combat sports events, thereby at least partially overcoming the disadvantages mentioned above in currently available solutions.

[0008] Furthermore, in video production, particularly for combat sports events, multiple cameras capturing the same event from different viewpoints are frequently used to provide comprehensive coverage. Traditional methods for handling multiple video sequences are often based on manual intervention, posing a significant challenge to both real-time and post-event video production tasks. Typically, creating a compelling output video sequence from multiple input video sequences involves manual panning, tilting, and zooming of multiple cameras, video switching based on directorial choices, and extensive post-production work to select the best shots, remove redundant sequences, and edit the video into a coherent and engaging format.

[0009] In other words, video production for combat sports events traditionally requires a significant amount of manual labor, including camera operators, technicians, production controllers, and specialized equipment. Given the human and financial resources involved, such professional-grade production is typically limited to large-scale events with substantial budgets, and is not feasible for smaller tournaments, gym fight nights, or daily sparring training.

[0010] Furthermore, the traditional process is highly subjective and heavily reliant on the skill level of the director or video editor. Therefore, it doesn't guarantee perfectly sequenced video, as real-time decisions may not always lead to the optimal viewpoint selection. Additionally, identifying exciting moments in combat sports events, typically involving major moves such as strikes, kicks, or takedowns, is usually done manually. This process is not only tedious but also prone to overlooking crucial parts of the event.

[0011] While consumer-grade cameras and smartphones offer a cost-effective alternative for capturing video footage, they lack the ability to autonomously follow fast-paced action in a game, as well as the capacity to add statistics and highlight moments that are crucial aspects of the viewing experience. Furthermore, games and highlight moments are best captured when cameras are available from multiple angles, allowing switching to any angle that provides the best viewing experience at a given moment.

[0012] Therefore, automated methods are needed to handle the complexities associated with creating video sequences of combat sports events that are engaging and appealing to audiences.

[0013] Despite substantial progress in video production and processing, further improvements and innovations are still needed, particularly in the automation and optimization of video production for combat sports events. This context provides the background for this disclosure.

[0014] Therefore, the second technical problem on which this invention is based is to provide a method for automated video production of combat sports events, thereby at least partially overcoming the disadvantages mentioned above in currently available solutions. Summary of the Invention

[0015] The first problem is addressed by the subject matter defined in the independent claims. Advantageous modifications of embodiments of the invention are defined in the dependent claims, as well as in the description and drawings.

[0016] According to a first aspect of the invention, a method for automated evaluation of a combat sports event is provided. The method may be computer-implemented. The method may include receiving at least one input video sequence of the combat sports event. The at least one input video sequence may be captured by at least one video camera. The method may include processing the at least one video sequence. The processing steps may include automatically determining at least one region of interest (ROI) in the video sequence. Automatic determination may be performed by a target detection algorithm, particularly a machine learning-based target detection algorithm. The at least one ROI may be associated with a first fighter in the combat sports event, preferably with at least one body part of the first fighter. The processing steps may include automatically determining activity data of the first fighter's fighting activities relative to a second fighter in the combat sports event. The automatic determination of activity data may be performed based on the at least one ROI in the video sequence. The automatic determination of activity data may be performed by an activity tracking algorithm, particularly a machine learning-based activity tracking algorithm. Activity data may include strikes performed, particularly successful strikes by the first fighter. The method may include generating statistics of the combat sports event based on the activity data, preferably continuously generated throughout the entire duration of the combat sports event.

[0017] Combat sports or confrontational sports should be understood as contact sports that typically involve one-on-one combat. Combat sports events can in particular include boxing events. Other combat sports that can be applied to include Mixed Martial Arts (MMA), wrestling, kickboxing, taekwondo, Brazilian Jiu-Jitsu (BJJ), etc.

[0018] In combat sports events, the setting, in accordance with the rules and regulations of the sport, typically includes a designated action area, which may be referred to as the boxing ring or fighting zone. Contained within the boxing ring or fighting zone are the fighters who participate in the combat sports event. Usually, two or more fighters are involved, with the aim of competing against each other. Each fighter is an active participant engaging in physical actions relative to each other. Fighters may execute actions and / or techniques according to their training, tactical intentions, and the rules set by the respective combat sport.

[0019] Besides the fighters, another entity within the arena can be the referee, who can be assigned to oversee combat sports events (i.e., the fighters' combat actions). The referee's duties may include observing, making decisions based on combat actions, enforcing sports rules, issuing warnings or penalties for violations by fighters, and / or maintaining the fair and safe conduct of combat sports events. In cases of extreme physical condition or unforeseen events, the referee may exercise their powers to introduce a stoppage, halt an ongoing contest, announce the result, and / or ensure the safety of the fighters.

[0020] Each participant in a combat sports event, namely the fighter and the referee, can be an object of region of interest. As described in further detail below, the region of interest can therefore be, for example, the fighter or referee themselves, or a part of their body. In particular, the following examples of body parts may be relevant: a fighter's "fist" (which may be protected by gloves in sports such as boxing) serves as one of the primary points of contact for executing fighting moves against an opponent. A fighter's "forearm" and / or "elbow" are frequently involved in various fighting moves in combat sports, particularly for blocking incoming strikes or initiating offensive actions. Lower body parts of a fighter, such as the "foot" and / or "shin" and / or "knee," are also relevant in various combat sports, either as body parts that are not permitted to be struck, or as body parts used in fighting moves (e.g., in kick-based combat sports such as taekwondo or kickboxing). In some combat sports (such as boxing), a fighter's "head" can also be a point of contact for offensive or defensive fighting moves. The fighter's "torso" (which may include the fighter's chest and / or abdominal area) is generally considered a preferred target area for striking. Furthermore, in various combat sports, such as grappling sports like judo or wrestling, additional physical interactions may be relevant, for example, involving the fighter's "shoulder," "hip," "legs," and / or similar areas used for throws or pinning the corresponding opponent.

[0021] The method may include receiving at least one video sequence from the combat sports event. The capture of this input video sequence is facilitated and performed by one or more video cameras, which may be strategically positioned around the fighting area to ensure maximum coverage, thereby allowing for comprehensive input data for analysis. In other words, the method may be implemented using at least one video camera constituting one aspect of a hardware setup, which may also include a computing device that performs several steps to ensure thorough analysis of the video(s) and to generate high-quality output for an improved viewer experience. For example, one or more of the method steps according to the invention may be performed by a server and / or a cloud platform and / or a local computer placed near the combat sports event and / or by any other suitable computing device.

[0022] The term "video camera" can be understood as an imaging device configured to capture scenes to generate video sequences. Specifically, a video camera can be configured to capture sequences of images that produce the effect of motion in a rapid, continuous manner, thus generating a video sequence. Depending on the video camera's viewpoint, the captured video sequences include action scenes during combat sports events, recording one or more movements involving the video camera's angle of view and zoom factor, one or more facial expressions, and / or the like. Video cameras can be equipped with advanced technologies such as autofocus, image stabilization, high-definition capture capabilities, and a range of frame rate options for multi-functional capture.

[0023] The term "viewpoint" can be understood as the position of a corresponding video camera relative to the fighting area or boxing ring, preferably including the corresponding angle of view, more preferably including the corresponding zoom factor and / or other settings of the corresponding video camera. Thus, in other words, the viewpoint of a video camera can include the positioning and / or orientation of the corresponding video camera, and optionally other settings of the corresponding video camera.

[0024] The term "video sequence" can essentially be a collection of consecutive video frames or images that convey motion when played back. Within the tight temporal order of the images that make up a video sequence, each image can be referred to as a frame. These sequences of frames, captured by a video camera, may encapsulate not only the ongoing physical actions in a combat sport event, but also the fleeting dynamics and interactions between fighters. It can comprehensively capture the progress of a fight, including strikes, responsive dodges, referee actions, crowd behavior, and the overall atmosphere of the match. Individual video sequences can involve varying lengths, for example, influenced by the purpose for which the sequence is being analyzed.

[0025] The term "computer-implemented" refers to a computing device, that is, a device that uses computer systems or digital technologies to perform certain tasks or processes. In the context of this invention, it relates to the deployment of a set of algorithms and / or applications and / or digital tools used by a computer to perform the objectives of the method. Thus, a computing device can participate in various stages of the method, e.g., from initially receiving video sequences, to processing and analyzing these sequences, all the way to the final output generation. In particular, the computing device may include, but is not limited to: object detection (which can be used to determine regions of interest in the video sequence) and / or machine learning techniques (which can be used to interpret interactions within the regions of interest), and / or video editing (for compiling and outputting the desired video sequence).

[0026] The term "object detection algorithm" can be understood as an instance of computational processes employed to identify the presence of objects in images or frames of a video sequence, particularly to identify regions of interest. It can be a computer program or set of instructions that enables a computer system to distinguish and / or locate certain objects from a large amount of visual data that may otherwise be unclassified. As can be implemented in the context of this invention, an object detection algorithm works by scanning a video sequence and identifying segments where a predetermined "object" (particularly a "region of interest") is present. Examples may include fighters in a combat sport, the referee in a combat sport, and / or body parts of the fighters and / or the referee. An object tracking algorithm can analyze images or frames of a video sequence to determine visual patterns and / or characteristics consistent with possible predefined definitions of the object and / or region of interest.

[0027] In more advanced implementations, object detection algorithms can be machine learning-based, meaning they learn and / or evolve and / or are pre-trained from training datasets to identify targets and / or regions of interest. By learning from training data, machine learning-based algorithms can predict and / or identify targets and / or regions of interest in new, previously unseen video sequences. Furthermore, object detection algorithms can be configured to simultaneously track multiple targets and / or regions of interest, detect target overlap, identify occlusion, and / or even classify detected targets and / or regions of interest into different categories. These combined capabilities of object detection algorithms can contribute to the efficiency and reliability of the methods according to the present invention.

[0028] The term "machine learning" can be understood as a subset of artificial intelligence, involving the construction of algorithms that allow computer systems to learn from data and make predictions or decisions based on that data. One form of machine learning model that can be applied in this invention is a supervised learning model. These models—trained on labeled datasets—can be used to identify regions of interest. For example, Convolutional Neural Networks (CNNs), popular in image analysis tasks, can be employed to detect and locate fighters or specific body parts in each frame of a video sequence. As an alternative or addition to supervised learning models that require labeled datasets, unsupervised learning models (such as clustering algorithms and / or autoencoders) can be used to reveal hidden and / or occluded patterns or structures in unlabeled data. In the context of this invention, these can be employed to learn and model normal behavior or movement patterns of fighters and detect anomalies based on deviations from normal. Alternatively or additionally, semi-supervised learning models can be used, which combine the principles of both supervised and unsupervised learning. Given the large volume of video data, labeling each frame can be a daunting task. Semi-supervised learning models can learn from relatively small labeled datasets and large unlabeled datasets, thereby improving the efficiency of the training process. Alternatively or additionally, reinforcement learning models can be used. These models learn how to act based on feedback or reward signals. They can be used to train object detection algorithms or determine quality-of-view indicators by learning the optimal parameters or methods to maximize audience satisfaction or engagement through trial and error. Furthermore, deep learning models, which are subsets of machine learning that represent complex, hierarchical data abstractions, can also be applied. For example, recurrent neural networks (RNNs) or long short-term memory (LSTM) networks—which are efficient in processing sequential data—can be employed to analyze the movement sequences of a fighter across frames. Machine learning models can be designed as classifiers. By using machine learning models, the steps of the methods can be performed with greater accuracy and efficiency.

[0029] It can be specified that the target tracking algorithm is based on a machine learning model trained through supervised or semi-supervised machine learning, where the training data is a pre-labeled dataset. The training data can represent the underlying knowledge used in the machine learning model, comprising a set of instances, each labeled with relevant tags and / or categories describing aspects of interest (e.g., for video sequences and / or for regions of interest). The training data can cover a database of video sequences from combat sports events, labeled (e.g., frame-by-frame) with relevant information. For example, relevant targets and / or regions of interest—which can be fighters, referees, or specific body parts of fighters or referees, such as the head, fist, foot, etc.—can be labeled.

[0030] The basic design of the machine learning model, as described above, includes various implementation options and, with necessary modifications, is applicable to all other machine learning models mentioned in the following disclosure of this invention.

[0031] The term "region of interest" can be understood as a selected subset or region within the entire dataset that is the focus of further analysis or processing. In the context of video processing and specifically for this invention, a region of interest in a video sequence can refer to a segment of the video sequence in which a particularly relevant target, such as a fighter, referee, and / or one or more of their body parts, is located, and / or in which a particularly relevant action related to a combat sport occurs. Given the dynamic nature of combat sports, the region of interest may move within the video sequence and is preferably tracked within the video sequence. In other words, the region of interest can be defined variably in time and space. For example, during a boxing match, if the region of interest is determined to be around the boxer's fist, it will constantly change in position and size as the fist moves and changes shape throughout the match. Therefore, the region of interest can be a cropped view or portion of the video sequence, i.e., a portion of a video image frame.

[0032] The term "combat activity" can be understood to include any physical actions performed by participants, particularly fighters, during a combat sports event. This typically includes offensive actions (such as striking, kicking, throwing, and grappling) as well as defensive actions (such as blocking, parrying, and dodging). Additionally, it can include any body movement as well as gestures and sounds. For example, in boxing, combat activity could encompass all the jabs, straight punches, hooks, and uppercuts thrown by the boxer, their footwork, and their defensive actions such as blocking and diving.

[0033] The term "combat activity activity data" can be understood as quantitative or qualitative data derived from identified and tracked combat activities performed during combat sports events. This includes information about the type, frequency, success rate, and / or intensity of combat activities. For example, in judo, activity data could include the number and type of throws attempted, the number of successful throws, the duration of hold, escapes from hold, and other relevant combat maneuvers.

[0034] The term "activity tracking algorithm" can be understood as a computational algorithm, typically based on machine learning, designed to monitor, identify, and track combat activities in video sequences of combat sports events. For example, this could be an algorithm trained to detect takedowns by wrestlers in a wrestling match, track their execution over time, and record relevant data about their occurrence.

[0035] The term "statistics" can be understood as processed output data derived from tracked activity data. This data is typically in the form of understandable statistical information that summarizes, describes, and / or interprets the overall performance of one or more fighters in a combat sport. In Taekwondo, statistics might include the total number of kicks landed, the percentage of successful blocks, or the frequency of each type of kick used. In boxing, statistics may include statistics based on boxing rules and regulations, particularly the power and clarity of strikes, and / or defensive work, and / or ring control, and / or effective attacks.

[0036] The term "duration" can be understood as the total length of time within which a combat sports event and / or a specific round within the event takes place. For example, in the context of a boxing match, duration can refer to the entire match, a single round within the match, or even a specific segment, such as the time a boxer spends on offense and defense.

[0037] First, the automated evaluation of combat sports events provided by the method according to the invention offers an innovative way to analyze combat sports events, supplementing and / or supporting human judgment with machine-based evaluation to eliminate potential bias and reduce errors. Second, using video sequences as input allows for the diversity of source data. It can accept video from any type of camera positioned anywhere around the fighting zone, thus enhancing the practical applicability of the method. Third, the automatic identification of regions of interest associated with a particular fighter ensures that no key events within the match are missed, thus creating a reliable foundation for all subsequent analyses. The inclusion of target detection algorithms (particularly machine learning-based algorithms) brings a fourth advantage, which is further described below. Fifth, if the method again uses machine learning in determining the activity data, machine learning-based activity tracking algorithms dissect the fighting activities in the regions of interest, outputting valuable information such as the type of action, their frequency, and the intensity or effectiveness of possible movements. A sixth advantage lies in the specificity of the activity data determination, which focuses on the fighting activities and the strikes performed. This focus enables the system to generate data particularly relevant to combat sports, making the output highly useful for accurate performance evaluation. The seventh advantage stems from the generation of statistical data, where isolated strings of activity data are synthesized into coherent, actionable insights into combat sports events. This could involve a range from the frequency of successful strikes to the fighter's attack level, thus providing a comprehensive overview of the combat event. Finally, the potential for continuous evaluation and statistical data generation throughout the event marks the eighth advantage. This facilitates real-time performance monitoring and allows for dynamic assessments that can guide tactics and decision-making, providing an unprecedented level of engagement for both athletes and spectators. The method provided by this invention can impactfully transform how combat sports events such as boxing, judo, or wrestling are evaluated.

[0038] It can be specified that the at least one input video sequence is multiple input video sequences. In this case, processing steps can be performed on each of the multiple video sequences. The multiple input video sequences can be captured by multiple video cameras.

[0039] Each video sequence can provide a unique view of a combat sport event created from its respective viewpoint (e.g., position, angle, and / or zoom factor relative to the event). Parallel processing of the video sequences provides a more detailed record of the event, leaving fewer blind spots and measuring more nuances of the activities unfolding on the field. Instead of acting on a single video camera source, the object detection algorithm and subsequent method steps of this invention can process each video sequence individually. By doing so, the technique allows for capturing a comprehensive picture of the event from multiple viewpoints, thereby maximizing the value of the generated statistics.

[0040] In particular, parallel processing of more than one video camera source is advantageous when determining the positioning and movement of fighters. In rapidly evolving events such as combat sports, a single moment can have a significant impact on the audience's understanding and appreciation. Using a single video source, the opportunity to capture such a moment is inherently limited by the given viewpoint and frame. Furthermore, the use of multiple video cameras contributes to a higher level of fault tolerance. In the event of a technical malfunction, failure, or any other problem with one camera, the system will not lack data to process; the sources from the remaining cameras continue to provide input, thus ensuring uninterrupted execution. Moreover, using more than one video camera is advantageous to improve the stability of the method according to the invention.

[0041] It can be specified that automatic determination of activity data is performed without prior posture determination of the fighter. Additionally or alternatively, automatic determination of activity data can be performed without defining and / or detecting keypoints for the fighter's body parts. Posture determination can be understood as a process in computer vision to predict the transformation of a target from a user-defined reference posture, given an image or 3D scan. In the case of human posture estimation, major joints such as knees, elbows, shoulders, and wrists typically represent keypoints, upon which a "stick figure" representation of the person is generated. Subsequent processing operates only on the stick figure model, not on the original image data.

[0042] Additionally or alternatively, automatic determination of activity data can be performed (directly) based on image frames, particularly image frame pixels, of the at least one region of interest in the video sequence. Methods that do not use pose determination and / or keypoints for determining combat activities, particularly human-to-human contact detection, are particularly useful in terms of required processing resources and detection accuracy. Instead of relying on pose estimators and / or keypoints of the fighter's body parts, direct machine learning-based operations, particularly direct neural network-based classification, can be performed on the video sequence, particularly video frames of the video sequence, particularly the region of interest. To overcome the occlusion problem as further described below and the confusion of the fighter's and / or referee's identities, in the proposed method, a set of labels, such as visual markers and / or other suitable indicators, can be applied, allowing the corresponding machine learning(s) to learn to detect and classify combat activities and correctly associate them with the corresponding fighters, even if the video sequence includes severe occlusion (e.g., overlap between the body parts of (one or more) fighters, (one or more) referees, and / or (one or more) fighters and / or (one or more) referees). Adding labels to (one or more) regions of interest and / or video sequences is described in further detail below.

[0043] It can be specified that the method is performed based on multiple machine learning models, wherein performing the method includes: utilizing a sequence of at least two machine learning models, and / or wherein performing the method includes: utilizing at least three machine learning models, wherein the outputs of at least two of the at least three machine learning models are used as inputs to at least one of the at least three machine learning models.

[0044] Including multiple machine learning models in the method according to the invention constitutes a significant enhancement. The method of the invention enables the program to leverage the specialized advantages of various machine learning models, resulting in collaborative, multifaceted operation at runtime. In particular, by using a strategic sequence of machine learning models, a unique mosaic of the capabilities of each model emerges, which collectively improves the output quality of the method.

[0045] For example, a machine learning model can focus on interpreting an input video sequence to determine regions of interest. This model can be specialized for target detection and tracking, thus ensuring accurate observation of the most critical segments involving the fighters. Another machine learning model can focus on evaluating combat activities, i.e., automatically determining activity data of the first fighter's combat activities relative to a second fighter in the combat sport event, particularly the activity data of strikes performed, based on said at least one region of interest in the video sequence. This is to evaluate the output of the first machine learning model, i.e., to determine the activity data of combat activities included in one or more regions of interest. This model can excel at providing a deep foundation for performing the following steps: preferably generating statistics throughout the entire duration of the combat sport event.

[0046] One or more of the machine learning models can be implemented as classifiers. For example, an object detection algorithm can classify different regions in a video sequence as regions of interest or not. More sophisticated classifiers can even classify the type of object (i.e., body parts and / or similar objects). Additionally or alternatively, classifiers may also potentially help determine the activity data of combat activities, classifying them into categories such as "successful," "unsuccessful," or "high attack level," "medium attack level," "low attack level," or "high ring control," "medium ring control," "low ring control," or "strong defensive activity," "weak defensive activity," or "heavy punch," "weak punch," "clear punch," "unclear punch," or any other suitable level category based on certain predefined criteria or thresholds. For combat sports as boxing events, the levels and / or criteria and / or thresholds can be considered a "10-Point Must System," which includes the criteria for judging boxing matches and / or includes the possible outcomes of boxing matches.

[0047] The focus of this concept is on achieving output of generally good quality while simultaneously facilitating real-time operation with low processing resources. Machine learning models can be advantageously selected and ranked based on their individual and combinatorial capabilities. Examples of other combinatorial machine learning models are given in further disclosure below.

[0048] It can be stipulated that at least one region of interest is automatically determined based on a deterministic machine learning model.

[0049] As described above, identifying the region of interest and, based on that, determining the combination of activity data is particularly advantageous and provides a powerful concept for improving the quality of expected statistics for corresponding combat sports events. These statistics can support judges in making their decisions and / or support audiences in better understanding and judging combat sports events themselves.

[0050] Regions of interest—as identified by object detection algorithms—encompass the most critical fighting movements in combat sports. An example of this is seen in boxing, where rapid fist movements are particularly relevant. Here, the object detection algorithm might define the boxer's fist, or the entire body or upper body including the fist, as a region of interest, effectively encapsulating the segment of a video sequence where the boxer (in this case, the first fighter) is striking. This approach ensures that every significant moment or visually most compelling action is not lost in the noise but is processed with full attention. Combined with this, defining the concept of combat activity is a mechanism for extracting content included within the corresponding region of interest.

[0051] It is important to note that, when considering the principles of the invention, a region of interest (ROI) in a video sequence can be a specific subsequence of that video sequence. This subsequence can include portions of video frames or images and can also span temporal phases of the video sequence. Therefore, the ROI is not limited to isolated frames or moments but can be extended to include a set or sequence of frames that collectively contribute to capturing the dynamics and continuity of the fighter's movements. Using machine learning models to determine the ROI and to determine the activity data is particularly advantageous and can provide enhanced output quality. In terms of implementation, these two tasks can be included in a single machine learning model. However, providing two separate machine learning models for these two tasks (where the output of, for example, the first machine learning model for determining the ROI serves as the input to, for example, the second machine learning model for determining the activity data) is particularly advantageous due to processing resources and for facilitating a real-time experience.

[0052] Therefore, it can be specified that the at least one region of interest in the video sequence is a subsequence of the video sequence that includes a time period of the video sequence and / or a cropped image portion. Furthermore, it can be specified that the at least one region of interest is marked by a bounding box representing a corresponding image frame portion of the video sequence.

[0053] The region of interest identified in a corresponding video sequence can extend beyond a single frame or moment, or alternatively, it can be a specific subsequence of the video sequence. This subsequence can contain both a time period from the main video sequence and a cropped portion of the image. For example, in a boxing match, such a subsequence might span several frames capturing key striking sequences or significant defensive maneuvers. This granular segmentation allows for the isolation of salient events within the match, enabling a more targeted analysis of the fighter's performance.

[0054] To delineate these regions of interest, bounding boxes can be used, which graphically encapsulate the associated video image frame portion representing the key action. For example, if the corresponding region of interest is the first fighter's fist, the sequence of the first fighter's fist from the moment of initiation to the point of impact can be encapsulated within the bounding box. Using such bounding boxes provides an accurate, visual way to identify and track each prioritized portion of the video sequence. The temporal approach captures the progress and outcome of combat activities, providing not only what but also dynamic insights into when and the subsequent results. This complexity of data capture leads to insightful statistical data generation. Furthermore, using cropped image portions and bounding boxes ensures efficient processing. By focusing on key regions, the system reduces the computational requirements for analyzing entire frames or extended sequences, resulting in lower resource usage and faster processing. Additionally, marking these regions of interest with bounding boxes provides a clearer visual representation of the focal area. This visual data aids subsequent processing stages and can be beneficial for manual review and evaluation of results, such as by referees in combat sports events.

[0055] It can be specified that each of the at least one region of interest is associated with a fighter, that is, with the first fighter or the second fighter; and Optionally, if the corresponding region of interest is associated with the first fighter, only the activity data of the first fighter's fighting activities are determined, i.e., the activity data of the second fighter's fighting activities are ignored.

[0056] To generate activity and statistics effectively and accurately, it is beneficial to associate each identified region of interest (ROI) with a specific fighter participating in a combat sport event. For example, in the context of a karate championship, the system might define a ROI around the first fighter's arm during a strike. In this case, the ROI is essentially associated with or linked to the first fighter. This clearly defined connection helps to accurately map the performed activity to the responsible fighter, thus eliminating any space for confusion or misattribution in the fast-paced action of combat sports.

[0057] Furthermore, the implemented method can be programmed to explicitly focus its attention on fighters associated with a defined region of interest. This means that if a specific region of interest is associated with a first fighter, the system will specifically track and identify that fighter's activity data. Continuing with the previous embodiment, if the region of interest associated with the first fighter is a striking arm, the system will accordingly ignore the actions of the second fighter. This focused analysis can provide detailed, precise performance data about individual fighters, rather than a general overview, which can help referees make more informed decisions.

[0058] The value of this focused assessment extends to a range of technical advantages. First, it ensures that actions or movements are accurately attributed to the correct fighter, eliminating the chance of misidentification and providing an unbiased measure for judgment or evaluation. Second, it simplifies the process of determining activity data and / or generating statistics by limiting the focus to the activities of a fighter specifically associated with the area of ​​interest, and reduces computational requirements. Third, the concept improves the accuracy of motion tracking and activity data determination, thereby facilitating the generation of detailed, highly accurate statistics for each fighter.

[0059] It can be specified that at least one region of interest is tracked within the video sequence based on at least one feature included in the video sequence. The at least one feature may be a feature of content included in the video sequence, and / or may be an intentionally introduced marker.

[0060] The concept of using one or more features within a video sequence facilitates the observation of the identified region of interest. In other words, features can be supporting elements for performing processing, particularly for determining the region of interest(s).

[0061] For example, in a boxing match, a characteristic could be the color of the fighter's gloves. Therefore, this characteristic can serve as a guiding reference for the system to track the movement of that body part (in this embodiment, the fighter's gloves) in consecutive frames or segments of a video sequence, thereby ensuring that important movements are accurately monitored throughout the match.

[0062] By using features to track regions of interest, actions and movements can be consistently and accurately tracked throughout a video sequence. Whether fighters move quickly or subtly, feature tracking ensures that every movement is captured and analyzed. Furthermore, the use of features keeps the region of interest in focus regardless of the complexity of the movement, the pace of the combat sports event, or changes in camera perspective. Even when multiple fighters are interacting at close range or when camerawork is rapidly changing, the feature can stably maintain tracking on a specified area. Moreover, this technique simplifies the automatic determination of activity data. Therefore, feature-based approaches can handle the dynamic and unpredictable nature of combat sports, effectively managing even more complex situations such as sudden changes of direction, high-speed movements, or overlapping movements, thereby enhancing the overall reliability and robustness of the system.

[0063] It can be specified that preprocessing of one or more of the at least one video sequence is performed before processing the video sequence, in order to reduce the processing load required to perform the processing. Alternatively or additionally, preprocessing is performed based on a preprocessing machine learning model.

[0064] The concept of performing a preprocessing stage before initiating core video sequence processing introduces several technical advantages. The preprocessing stage primarily prepares and / or refines (one or more) the video sequence for subsequent stages of the method. This can involve activities such as noise filtering, adjusting lighting or contrast, masking and / or padding irrelevant portions of the sequence, introducing labels into the video sequence, marking portions of the video sequence as key combat actions, and / or reducing resolution or frame rate for simplification. These adjustments help reduce redundant and / or irrelevant information, thereby streamlining the input that will be processed by subsequent stages of the method, thus reducing, for example, the demand on processing resources.

[0065] Enriching preprocessing with dedicated machine learning models can further improve the efficiency of this stage. Preprocessing machine learning models can learn to effectively detect and filter redundant and / or irrelevant information, identify relevant parts of video sequences, and optimally calibrate the input video sequences for subsequent tasks.

[0066] A significant advantage of this setup is the reduction in processing load. It alleviates the burden on the remaining stages of the process, thus providing a well-processable foundation for object detection algorithms and other tasks. Furthermore, reduced computational resources and processing time improve overall efficiency. This advantage becomes particularly valuable in real-time and / or live streaming scenarios, where minimizing the time spent from capture to broadcast is crucial. Moreover, the preprocessing step significantly improves the performance of object detection algorithms by providing a clearer, simpler input video sequence, reducing errors, and increasing accuracy.

[0067] It can also be specified that preprocessing includes: splitting video sequences by time period and / or cropping video sequences by image portion.

[0068] Splitting and / or cropping video sequences is highly beneficial. Splitting a video sequence refers to the temporal division of the input data. This involves dividing the entire length of the video sequence into smaller time segments and / or blocks. This can be particularly advantageous when processing long-duration sporting events, simplifying the processing of the video data and optimizing processing time.

[0069] Video sequence cropping can refer to the spatial segmentation of the input data. It's very similar to cropping a photo to focus on a specific object or viewpoint. Figure 1 Similarly, video cropping can involve selecting relevant or interesting parts of a video frame and ignoring the rest.

[0070] This helps to manage computational resources even more effectively. Working on smaller time intervals and focused portions of the image reduces the amount of data to be processed in each iteration step, thus requiring less memory and processing power. Furthermore, segmenting by time and space can lead to a significant reduction in processing time.

[0071] It can be specified that preprocessing includes associating at least one tag with the at least one region of interest and / or video sequence, wherein the at least one tag helps to associate the region of interest with the first fighter, and / or wherein the at least one tag provides additional information that can be used to determine activity data (especially statistics).

[0072] Tags can serve two main purposes: first, to facilitate connections between areas of interest and the first or second fighter or referee; second, to provide specific additional information that can enhance activity data (especially statistics). Tags can essentially be used as predefined categorical support information or markers that help streamline the extraction of relevant combat activities and statistics.

[0073] Suitable tags can be any kind of tags that can be detected and processed by subsequent steps of the method according to the present invention (i.e., the step of processing the at least one video sequence).

[0074] For example, if a corresponding region of interest (ROI) is labeled with a tag indicating that the corresponding ROI should be associated with the first fighter, then for that ROI, only the first fighter's activity can be considered to determine activity data and / or statistics. Specifically, if the first fighter successfully swings and strikes the second fighter's face, this data can be included in the activity data. However, if the second fighter successfully swings and strikes the first fighter's face, this data may not be included in the activity data, i.e., this activity is ignored. In other words, using labels to associate ROIs with specific fighters provides clarity in the data analysis process, ensuring that actions are correctly attributed to the correct fighter.

[0075] Furthermore, for example, if the corresponding regions of interest are marked with labels indicating which fighter is the first fighter and which fighter is the second fighter, this can be particularly useful for subsequent steps of the method according to the invention.

[0076] Providing labels advantageously offers a seamless and accurate transition to subsequent processes such as combat activity and statistical data determination. They can be calculated and assigned during the detector and / or tracker stages and added to the classifier input during the generator stage. Furthermore, labels can also be used to encapsulate and provide additional information, thereby enriching the dataset for more comprehensive analysis. Moreover, by providing labels (which offer additional information), the method according to the invention can generate more accurate and detailed statistics, thus improving the quality of the evaluation.

[0077] The process may also include automatically determining a viewing quality metric for at least one identified region of interest in the video sequence. The viewing quality metric may be determined, at least in part, based on the positional relationship between the first and second fighters in a combat sports event and / or the viewpoints of the corresponding video cameras, wherein the determination of the viewing quality metric is preferably performed based on a viewing evaluation machine learning model. When multiple cameras are available, such viewing quality estimation can also be used in strike counting. This can occur during the statistical data fusion stage, where certain viewpoints may be more advantageous than others for a given metric. Accordingly, if multiple cameras from different angles are running, one of the cameras with the best field of view can be selected for the specific metric being measured.

[0078] As an alternative or additional method to automatically determining viewing quality metrics based at least in part on the positional relationship between the first and second fighters in a combat sports event, the automatic determination of viewing quality metrics may be based at least in part on the positional relationship between at least one of the first and second fighters and the referee. Alternatively or additionally, the automatic determination of viewing quality metrics may be based at least in part on the positional relationship between at least one of the first and second fighters and the corresponding video camera capturing the corresponding video sequence. Alternatively or additionally, the automatic determination of viewing quality metrics may be based at least in part on the positional relationship between the referee and the corresponding video camera capturing the corresponding video sequence. Alternatively or additionally, the automatic determination of viewing quality metrics may be based at least in part on the positional relationships between body parts of the first fighter and / or between body parts of the second fighter.

[0079] Preferably, the automatic determination of viewing quality metrics can be based at least in part on the positional relationship between the first and second fighters in a combat sports event, and further on the positional relationship between at least one of the first and second fighters and the corresponding video camera capturing the corresponding video sequence. More preferably, the automatic determination of viewing quality metrics can be based at least in part on the positional relationship between the first and second fighters in a combat sports event, and further on the positional relationship between at least one of the first and second fighters and the corresponding video camera capturing the corresponding video sequence, and further on the positional relationship between body parts of the first fighter and / or between body parts of the second fighter.

[0080] It can also be specified that if processing is performed on multiple video sequences that display the same specific fighting activity in at least one corresponding region of interest, the selection of at least one reliable video sequence that includes the corresponding region of interest displaying the specific fighting activity is based on a viewing quality metric, wherein the automatic determination of activity data is performed based on this selection.

[0081] In combat sports, one of the major challenges that can affect the quality and clarity of video sequences is occlusion. This inevitably impacts evaluation, namely, the determination of activity data and / or the generation of statistics. The term "occlusion" in video processing and object detection refers to the partial or complete obstruction of one object by another in a video sequence. This phenomenon can occur in combat sports events, particularly when fighters overlap or overlap specific body parts in the video sequence, resulting in obstructed views from certain camera viewpoints. Furthermore, this phenomenon can occur if different body parts of one fighter overlap, resulting in self-occlusion. This is primarily due to fighters being in continuous movement, close contact, or engaging in offensive or defensive combat actions. Occlusion can vary based on the relative positioning of the fighters (especially relative to each other) and the viewpoint of the corresponding video cameras. In some cases, when fighters are side-by-side, from certain camera angles, one fighter may completely block the view of another. In other situations, when fighters are engaging in combat, body parts such as fists may obscure the view of another fighter's body, resulting in partial occlusion. Occlusion in combat sports presents significant challenges when applying video processing and / or target detection, particularly for performance evaluation.

[0082] The term "viewing quality metric" should be understood within the context of occlusion as described above. Therefore, a viewing quality metric can essentially quantify the potential content quality of a region of interest within a video sequence. Content quality can be closely related to the overall relevance of the region of interest within the corresponding video sequence, thus indicating whether the corresponding video sequence should be included in the output video sequence. Furthermore, as described above, content quality can be closely related to the degree of occlusion and / or self-occlusion. For example, occlusion may occur if the referee obstructs the view of the corresponding video camera, and / or if the fighter's position relative to the corresponding video camera is not optimal. The angle at which each fighter's side is visible in the video sequence may generally include higher viewing quality than the angle at which one fighter's front and another fighter's back are visible. An aspect that can indicate whether occlusion is significant can be the angle between the midpoint of a line virtually drawn from the corresponding video camera position and a line virtually drawn between the centers of mass of each fighter. If the angle between the two lines is 0°, the viewing quality can be low, while 90° can indicate a viewing angle with high viewing quality. In conjunction with this, to use a more audience-dependent term, viewing quality metrics can reveal the level of audience interest expected in regions of interest within a given video sequence.

[0083] Viewing quality metrics can be binary values ​​that are either 0 or 1, or values ​​defined within a predetermined scale, such as a range from 0 to 1, or any other suitable value that includes relevant information about viewing quality. When the viewing quality metric is a range of values, thresholds that constitute the boundary between "usable" and "unusable" viewing quality metric values ​​can be defined. Therefore, viewing quality metrics can carry, indirectly or directly, and in particular quantify, information about the degree of occlusion.

[0084] One aspect that helps determine the viewing quality metric is the positional relationship between the first and second fighters in a combat sports event. More specifically, this positional relationship can be based on the viewpoint relative to a virtual line drawn between the centers of mass of the two fighters' bodies. If this angle is measured at 90°, the view captures the fighters and the fighting action from a side-to-side perspective. Therefore, in this case, with minimal obstruction to the field of vision, the visualization and tracking of movement can be generally clear and detailed, resulting in visibility to the viewer. This can correspond to a high value for the viewing quality metric.

[0085] Conversely, if the angle is measured at 0°, the view is aligned exactly behind or in front of the fighters. Therefore, one fighter can severely obscure another, significantly blurring the view of their movement, actions, and expressions. This lack of visibility can correspond to a low value for the viewing quality metric. Consequently, even if relevant fighting actions occur, the viewing quality may not be good enough to use the corresponding scene in the region of interest as a basis for performance evaluation (i.e., for determining and / or generating statistical data for performance activities).

[0086] The combination of determining viewing quality metrics for regions of interest (ROIs) and determining activity data is particularly advantageous. By appropriately determining the viewing quality metrics for ROIs, the most relevant combat actions (such as swings and strikes) can be identified, including quality ratings that express how well the corresponding ROIs are visible within the corresponding video sequence. Therefore, appropriate ROIs and video sequences can be selected based on viewing quality metrics to facilitate the process of determining activity data for the combat sports event. In other words, viewing quality metrics establish a highly suitable basis for determining activity data, enabling the efficient and accurate generation of statistical data. For purposes of better understanding, it is mentioned here (effective throughout the disclosure of this invention) that a ROI of a video sequence can be a subsequence of the video sequence that contains that ROI. In particular, such a subsequence can include portions of video images or frames and / or time periods of the video sequence.

[0087] For example, in a boxing match scene, if the camera angle or the relative positioning of the boxers causes one boxer to obstruct the view of another's striking action, the viewing quality metric associated with that striking sequence will likely be low. Conversely, if the same striking action occurs with minimal or no obstruction in different input video sequences, thus providing clear visibility of the strikes within the video sequence, the viewing quality metric for that sequence will be high. This automatically guides the determination of activity data and / or the generation of statistics by using high-quality input data selected based on viewing quality metrics. This advantageously improves the quality of the generated statistics.

[0088] It can be specified that determining activity data and / or generating statistics is performed, which includes: performing a dual activity avoidance algorithm to avoid dual activity tracking, wherein preferably, the dual activity avoidance algorithm is based on non-maximum suppression. In addition to or as an alternative to non-maximum suppression (which may occur within one or more detectors), it can also be temporal non-maximum suppression used when there are signal outputs from one or more classifiers from multiple different cameras. Conceptually, the two techniques are similar, although one can be applied to 2D images while the other can be applied to time series of temporal signals.

[0089] Determining activity data—which incorporates a technique called nonmaximum suppression to circumvent problems associated with dual activity tracking—is particularly beneficial. Nonmaximum suppression is a technique in computer vision tasks, especially object detection, used to reduce multiple overlapping bounding boxes to a single bounding box that accurately defines the location of a single object. In object detection, the same object is often identified multiple times with slightly different bounding boxes. This leads to clustering of boxes around the same object, which can result in overcounting or representing a single object as multiple objects. Nonmaximum suppression addresses this problem by considering only the bounding box with the highest confidence score while ignoring all other overlapping boxes, thus suppressing nonmaximums.

[0090] For example, in the context of a boxing match, the system is using multiple machine learning models to identify and track the regions associated with the fighter's fists, generating multiple bounding boxes around the fist. Without nonmaximum suppression, each of these boxes might be incorrectly counted as a separate hit. However, applying nonmaximum suppression suppresses all boxes except those with the highest confidence score, effectively preventing duplicate counting of the same hit.

[0091] Therefore, by applying nonmaximum suppression, the method according to the invention can avoid inappropriate representation of actions and prevent overestimation of action counts, thereby improving the accuracy and reliability of automated evaluation of combat sports events. The integration of nonmaximum suppression during the activity data determination phase provides several significant advantages. It ensures that actions are not inaccurately double-counted or overestimated during the evaluation process, resulting in a more accurate representation of the fighter's performance. Furthermore, the application of nonmaximum suppression simplifies subsequent analysis by reducing overlap and redundancy in the tracked activities. This reduces the computational load and accelerates the overall evaluation process.

[0092] In object detection tasks, alternative methods to non-maximum suppression can be used to avoid double activity tracking. Mean-drift clustering involves machine learning algorithms that use clustering analysis. Mean-drift clustering is designed to find and measure “blobs” in a smooth density of samples. In the context of this invention, it can be employed to identify groups of similar activity data, thereby effectively locating the densest (and therefore most likely accurate) activity data in each group to prevent overcounting. Soft-Non-Maximum Suppression—a variant of non-maximum suppression—does not remove any bounding boxes but instead reduces their confidence scores based on their overlap with other detection boxes. This improves the overall performance and accuracy of object detection, especially for overlapping objects, while maintaining data diversity.

[0093] When multiple video sequences are received, the method may include synchronizing the multiple video sequences. Synchronization may be performed based on combat activities (especially combat activities occurring in a region of interest). Alternatively or additionally, synchronization may be performed based on determining the audio time offset between video sequences captured by different video cameras. Determining the audio time offset is an efficient method that requires relatively low processing resources.

[0094] According to a second aspect of the invention, there is a provision for using at least two machine learning models to perform the method according to the first aspect of the invention. It may also be specified that the output of at least one machine learning model can be used as the input of at least one different machine learning model.

[0095] According to a third aspect of the present invention, a data processing apparatus, preferably a computer device and / or a server and / or a cloud platform, is provided, including means for performing the method according to the first aspect of the present invention.

[0096] According to a fourth aspect of the invention, the use of a plurality of video cameras in a method according to a first aspect of the invention is provided. The video cameras can be deployed around combat sports events.

[0097] According to a fifth aspect of the invention, a system is provided that includes a plurality of video cameras according to a fourth aspect of the invention and a data processing device according to a third aspect of the invention, wherein the system is configured to perform a method according to a first aspect of the invention.

[0098] According to a sixth aspect of the invention, a computer program or a computer-readable medium having a computer program stored thereon is provided, the computer program including instructions that, when executed by a computer, cause the computer to perform a method according to a first aspect of the invention.

[0099] All features, technical implementation details, and advantages described with respect to any aspect of the invention herein are readily applicable to any other aspect of the invention, with necessary modifications, and vice versa.

[0100] The second problem is addressed by the topics defined in the following aspects.

[0101] According to a first aspect of the invention, a method for automated video production of combat sports events is provided. The method may be computer-implemented. The method may include receiving at least one input video sequence of a combat sports event captured by at least one video camera. The method may include processing the at least one video sequence. The processing steps may include automatically determining at least one region of interest (ROI) in the video sequence. Automatic determination may be performed by an object detection algorithm, particularly a machine learning-based object detection algorithm. The at least one ROI may be associated with a first fighter in the combat sports event, preferably with at least one body part of the first fighter. The processing steps may include automatically determining a viewing quality index for the determined at least one ROI. The automatic determination of the viewing quality index may be based at least in part on the positional relationship between a first fighter and a second fighter in the combat sports event. The method may include generating an output video sequence based at least in part on the determined viewing quality index.

[0102] As an alternative or additional method to automatically determining viewing quality metrics based at least in part on the positional relationship between the first and second fighters in the combat sports event, the automatic determination of viewing quality metrics may be based at least in part on the positional relationship between at least one of the first and second fighters and the referee. Alternatively or additionally, the automatic determination of viewing quality metrics may be based at least in part on the positional relationship between at least one of the first and second fighters and the corresponding video camera capturing the corresponding video sequence. Alternatively or additionally, the automatic determination of viewing quality metrics may be based at least in part on the positional relationship between the referee and the corresponding video camera capturing the corresponding video sequence. Alternatively or additionally, the automatic determination of viewing quality metrics may be based at least in part on the positional relationships between body parts of the first fighter and / or between body parts of the second fighter.

[0103] Preferably, the automatic determination of viewing quality metrics can be based at least in part on the positional relationship between the first and second fighters in the combat sports event, and further on the positional relationship between at least one of the first and second fighters and the corresponding video camera capturing the corresponding video sequence. Even more preferably, the automatic determination of viewing quality metrics can be based at least in part on the positional relationship between the first and second fighters in the combat sports event, and further on the positional relationship between at least one of the first and second fighters and the corresponding video camera capturing the corresponding video sequence, and further on the positional relationship between body parts of the first fighter and / or between body parts of the second fighter.

[0104] Combat sports or confrontational sports should be understood as contact sports that typically involve one-on-one combat. Combat sports events are particularly, especially, boxing events. Other combat sports that can be applied to include Mixed Martial Arts (MMA), wrestling, kickboxing, taekwondo, Brazilian Jiu-Jitsu (BJJ), etc.

[0105] In combat sports events, the arena, in accordance with the rules and regulations of the sport, typically includes a designated action area, which may be referred to as the boxing ring or fighting zone. Contained within the boxing ring or fighting zone are the fighters who participate in the combat sports event. Usually, two or more fighters are involved, with the aim of competing against each other. Each fighter is an active participant engaging in physical actions relative to each other. Fighters may execute actions and / or techniques according to their training, tactical intentions, and the guidelines set by the respective combat sport.

[0106] Besides the fighters, another entity within the arena can be the referee, who can be assigned to oversee combat sports events (i.e., the fighters' combat actions). The referee's duties may include observing, making decisions based on combat actions, enforcing sports rules, issuing warnings or penalties for violations by fighters, and / or maintaining the fair and safe conduct of combat sports events. In cases of extreme physical condition or unforeseen events, the referee may exercise their powers to introduce a stoppage, halt an ongoing contest, announce the result, and / or ensure the safety of the fighters.

[0107] Each participant in a combat sports event, namely the fighter and the referee, can be an object of region of interest. As described in further detail below, the region of interest can therefore be, for example, the fighter or referee themselves, or a part of their body. In particular, the following examples of body parts may be relevant: a fighter's "fist" (which may be protected by gloves in sports such as boxing) serves as one of the primary points of contact for executing fighting moves against an opponent. A fighter's "forearm" and / or "elbow" are frequently involved in various fighting moves in combat sports, particularly for blocking incoming strikes or initiating offensive actions. Lower body parts of a fighter, such as the "foot" and / or "shin" and / or "knee," are also relevant in various combat sports, either as body parts that are not permitted to be struck, or as body parts used in fighting moves (e.g., in kick-based combat sports such as taekwondo or kickboxing). In some combat sports (such as boxing), a fighter's "head" can also be a point of contact for offensive or defensive fighting moves. The fighter's "torso" (which may include the fighter's chest and / or abdominal area) is generally considered a preferred target area for striking. Furthermore, in various combat sports, such as grappling sports like judo or wrestling, additional physical interactions may be relevant, for example, involving the fighter's "shoulder," "hip," "legs," and / or similar areas used for throws or pinning the opponent.

[0108] The method may include receiving at least one video sequence from the combat sports event. The capture of this input video sequence is facilitated and performed by one or more video cameras, which may be strategically positioned around the fighting zone to ensure maximum coverage, thereby allowing for comprehensive input data for analysis. In other words, the method may be implemented using at least one video camera constituting one aspect of a hardware setup, which may also include a computing device performing several steps to ensure thorough analysis of the video(s) and to generate high-quality output for an improved viewer experience. For example, one or more of the method steps according to the invention may be performed by a server and / or a cloud platform and / or a local computer located near the combat sports event and / or by any other suitable computing device.

[0109] The term "video camera" can be understood as an imaging device configured to capture scenes to generate video sequences. Specifically, a video camera can be configured to capture sequences of images that produce the effect of motion in a rapid, continuous manner, thus generating a video sequence. Depending on the viewpoint of the video camera, the captured video sequences include action scenes during combat sports events, recording one or more movements, one or more facial expressions, and / or similar elements involving the video camera's viewpoint and zoom factor. Video cameras can be equipped with advanced technologies such as autofocus, image stabilization, high-definition capture capabilities, and a range of frame rate options for multi-functional capture.

[0110] The term "viewpoint" can be understood as the position of a corresponding video camera relative to the fighting area or boxing ring, preferably including the corresponding angle of view, more preferably including the corresponding zoom factor and / or other settings of the corresponding video camera. Thus, in other words, the viewpoint of a video camera can include the positioning and / or orientation of the corresponding video camera, and optionally other settings of the corresponding video camera.

[0111] The term "video sequence" can essentially be a collection of consecutive video frames or images that convey motion when played back. Within the tight temporal order of the images that make up a video sequence, each image can be referred to as a frame. These sequences of frames, captured by a video camera, may encapsulate not only the ongoing physical actions in a combat sport event, but also the fleeting dynamics and interactions between fighters. It can comprehensively capture the progress of a fight, including strikes, responsive dodges, referee actions, crowd behavior, and the overall atmosphere of the match. Individual video sequences can involve varying lengths, for example, influenced by the purpose for which the sequence is being analyzed.

[0112] The term "computer-implemented" refers to a computing device, that is, a device that uses computer systems or digital technologies to perform certain tasks or processes. In the context of this invention, it relates to the deployment of a set of algorithms and / or applications and / or digital tools used by a computer to perform the objectives of the method. Thus, a computing device can participate in various stages of the method, e.g., from initially receiving video sequences, to processing and analyzing these sequences, all the way to the final output generation. In particular, the computing device may include, but is not limited to: object detection (which can be used to determine regions of interest in the video sequence) and / or machine learning techniques (which can be used to interpret interactions within the regions of interest), and / or video editing (for compiling and outputting the desired video sequence).

[0113] The term "object detection algorithm" can be understood as an instance of computational processes employed to identify the presence of objects in images or frames of a video sequence, particularly to identify regions of interest. It can be a computer program or set of instructions that enables a computer system to distinguish and / or locate certain objects from a large amount of visual data that may otherwise be unclassified. As can be implemented in the context of this invention, an object detection algorithm works by scanning a video sequence and identifying segments where a predetermined "object" (particularly a "region of interest") is present. Examples may include fighters in a combat sport, the referee in a combat sport, and / or body parts of the fighters and / or the referee. An object tracking algorithm can analyze images or frames of a video sequence to determine visual patterns and / or characteristics consistent with possible predefined definitions of the object and / or region of interest.

[0114] In more advanced implementations, object detection algorithms can be machine learning-based, meaning they learn and / or evolve and / or are pre-trained from training datasets to identify targets and / or regions of interest. By learning from training data, machine learning-based algorithms can predict and / or identify objects and / or regions of interest in new, previously unseen video sequences. Furthermore, object detection algorithms can be configured to simultaneously track multiple targets and / or regions of interest, detect target overlap, identify occlusion, and / or even classify detected targets and / or regions of interest into different categories. These combined capabilities of object detection algorithms can contribute to the efficiency and reliability of the methods according to the present invention.

[0115] The term "machine learning" can be understood as a subset of artificial intelligence, involving the construction of algorithms that allow computer systems to learn from data and make predictions or decisions based on that data. One form of machine learning model that can be applied in this invention is a supervised learning model. These models—trained on labeled datasets—can be used to identify regions of interest. For example, convolutional neural networks (CNNs), popular in image analysis tasks, can be employed to detect and locate a fighter or specific body part in each frame of a video sequence. As an alternative or addition to supervised learning models that require labeled datasets, unsupervised learning models (such as clustering algorithms and / or autoencoders) can be used to reveal hidden and / or occluded patterns or structures in unlabeled data. In the context of this invention, these can be employed to learn and model normal behavior or movement patterns of fighters and detect anomalies based on deviations from normal. Alternatively or additionally, semi-supervised learning models can be used, which combine the principles of both supervised and unsupervised learning. Given the large volume of video data, labeling each frame can be a daunting task. Semi-supervised learning models can learn from relatively small labeled datasets and large unlabeled datasets, thereby improving the efficiency of the training process. Alternatively or additionally, reinforcement learning models can be used. These models learn how to act based on feedback or reward signals. They can be used to train object detection algorithms or determine viewing quality metrics by learning optimal parameters or methods to maximize audience satisfaction or engagement through trial and error. Furthermore, deep learning models, which are subsets of machine learning that represent complex, hierarchical data abstractions, can also be applied. For example, recurrent neural networks (RNNs) or long short-term memory (LSTM) networks—which are effective in processing sequential data—can be employed to analyze the movement sequences of a fighter across frames. Machine learning models can be designed as classifiers. By using machine learning models, the steps of the methods can be performed with greater accuracy and efficiency.

[0116] It can be specified that the target tracking algorithm is based on a machine learning model trained through supervised or semi-supervised machine learning, where the training data is a pre-labeled dataset. The training data can represent the underlying knowledge used in the machine learning model, comprising a set of instances, each labeled with relevant tags and / or categories describing aspects of interest (e.g., for video sequences and / or for regions of interest). The training data can cover a database of video sequences from combat sports events, labeled (e.g., frame-by-frame) with relevant information. For example, relevant targets and / or regions of interest—which can be fighters, referees, or specific body parts of fighters or referees, such as the head, fist, foot, etc.—can be labeled.

[0117] The basic design of the machine learning model, as described above, includes various implementation options and, with necessary modifications, is applicable to all other machine learning models mentioned in the following disclosure of this invention.

[0118] The term "region of interest" can be understood as a selected subset or region within the entire dataset that is the focus of further analysis or processing. In the context of video processing and specifically for this invention, a region of interest in a video sequence can refer to segments of the video sequence in which particularly relevant targets, such as fighters, referees, and / or one or more of their body parts, are located, and / or in which particularly relevant actions related to the combat sports event occur. Given the dynamic nature of combat sports events, the region of interest may move within the video sequence and is preferably tracked within the video sequence. In other words, the region of interest can be defined variably in time and space. For example, during a boxing match, if the region of interest is determined to be around the boxer's fist, it will constantly change in position and size as the fist moves and changes shape throughout the match. Therefore, the region of interest can be a cropped view of the video sequence, i.e., a portion of a video image frame.

[0119] In combat sports, one of the major challenges that can affect the quality and clarity of video sequences is occlusion. The term "occlusion" in video processing and object detection refers to the partial or complete obstruction of one object by another in a video sequence. This phenomenon can occur in combat sports events, particularly when fighters overlap or overlap specific body parts, resulting in obstructed views from certain camera viewpoints. Furthermore, this phenomenon can occur if different body parts of one fighter overlap, resulting in self-occlusion. This is primarily due to fighters being in a continuous, close-range, offensive or defensive combat maneuver. Occlusion can vary based on the relative positioning of the fighters (especially relative to each other) and the viewpoint of the corresponding video cameras. In some cases, when fighters are side-by-side, from certain camera angles, one fighter may completely block the view of another. In other cases, when fighters are engaging in combat, body parts such as fists may obscure the view of another fighter's body parts, resulting in partial occlusion. Occlusion in combat sports presents significant challenges when applying video processing and / or object detection.

[0120] The term "viewing quality metric" should be understood within the context of occlusion as described above. Therefore, a viewing quality metric can essentially quantify the potential content quality of a region of interest within a video sequence. Content quality can be closely related to the overall relevance of the region of interest within the corresponding video sequence, thus indicating whether the corresponding video sequence should be included in the output video sequence. Furthermore, as described above, content quality can be closely related to the degree of occlusion and / or self-occlusion. For example, occlusion may occur if the referee obstructs the view of the corresponding video camera, and / or if the fighter's position relative to the corresponding video camera is not optimal. The angle at which each fighter's side is visible in the video sequence may generally include higher viewing quality than the angle at which one fighter's front and another fighter's back are visible. An aspect that can indicate whether occlusion is significant can be the angle between the midpoint of a line virtually drawn from the corresponding video camera position and a line virtually drawn between the centers of mass of each fighter. If the angle between the two lines is 0°, the viewing quality can be low, while 90° can indicate a viewing angle with high viewing quality. In conjunction with this, to use a more audience-dependent term, viewing quality metrics can reveal the level of audience interest expected in regions of interest within a given video sequence.

[0121] Viewing quality metrics can be binary values ​​that are either 0 or 1, or values ​​defined within a predetermined scale, such as a range from 0 to 1, or any other suitable value that includes relevant information about viewing quality. When the viewing quality metric is a range of values, thresholds that constitute the boundary between "usable" and "unusable" viewing quality metric values ​​can be defined. Therefore, viewing quality metrics can carry, indirectly or directly, and in particular quantify, information about the degree of occlusion.

[0122] One aspect that helps determine the viewing quality metric is the positional relationship between the first and second fighters in a combat sports event. More specifically, this positional relationship can be based on a viewpoint relative to a virtual line drawn between the centers of mass of the two fighters' bodies. If this angle is measured at 90°, the view captures the side-to-side perspective of the fighters and the fighting action. Therefore, in this case, with minimal obstruction to the field of vision, the visualization and tracking of movement can be generally clear and detailed, resulting in visibility to the viewer. This can correspond to a high value for the viewing quality metric.

[0123] Conversely, if the angle is measured at 0°, the view is aligned exactly behind or in front of the fighters. Therefore, one fighter can severely obscure another, significantly blurring the view of their movement, actions, and expressions. This lack of visibility can correspond to a low value for the viewing quality metric. Consequently, even if relevant fighting actions occur, the viewing quality may not be good enough to include the corresponding scene from the region of interest in the corresponding video sequence in the output video sequence.

[0124] The combination of identifying regions of interest (ROIs) and determining viewing quality metrics for those ROIs is particularly advantageous. By appropriately identifying ROIs, the most interesting body parts (such as a boxer's fist in a boxing match) are selected, and thus the relevant fighting actions (such as a swinging strike) are determined accordingly. By appropriately determining the viewing quality metrics for the ROIs identified in one or more video sequences, a quality score is determined to express how well the respective ROIs are visible in the corresponding video sequence. For purposes of better understanding, it is hereby mentioned (and valid for the entire disclosure of this invention) that a ROI of a video sequence can be a subsequence of the video sequence that contains that ROI. In particular, such a subsequence can include portions of video images or frames and / or time periods of a video sequence.

[0125] The method introduced in this invention offers numerous technical advantages and has revolutionized video production for combat sports events. Starting with receiving at least one input video sequence of a combat sports event recorded by at least one video camera, this method represents a first step towards precise processing of combat sports events. The flexibility to process single or multiple video inputs expands the invention's ability to capture diverse views, providing comprehensive coverage of the event.

[0126] When multiple video sequences are received, the method may include synchronizing the multiple video sequences. Synchronization may be performed based on combat activities (especially combat activities occurring in a region of interest). Alternatively or additionally, synchronization may be performed based on determining the audio time offset between video sequences captured by different video cameras. Time-audio time offset determination is an efficient approach that requires relatively low processing resources.

[0127] The automation of this method offers significant advantages during the processing of one or more video sequences. Preferably, a machine learning-based object detection algorithm operates automatically, replacing manual tracking and detection with a more robust, efficient, and accurate alternative. It identifies regions of interest (and, if necessary, even specific body parts) associated with the first fighter involved in the event. This advanced object detection has great potential in capturing and highlighting dynamic moments in combat sports, thereby enhancing the communication of the sports narrative. Another important advantage offered by this method involves the automatic determination of viewing quality metrics. Viewing quality metrics help quantify the degree of occlusion and, therefore, the potential audience interest that specific segments of a video sequence may generate. Using the positional relationship between the first and second fighters as a contributing factor, these metrics take into account the spatial dynamics inherent in combat sports that significantly influence audience engagement, particularly the occlusion phenomena described above.

[0128] Perspective plays a crucial role in this determination and can be implemented using machine learning models. When the angle directly focuses the viewer on the combat actions between fighters, viewing quality is optimized, resulting in better visibility and less obstruction of the event. Conversely, severely obstructed views reduce viewing quality. Utilizing the automatic determination of viewing quality metrics, this method can intelligently guide the automated editing process. Key moments can be selected and highlighted, effectively conveying the excitement of combat sports to the audience.

[0129] Subsequently, after absorbing and analyzing key aspects of the input video sequence, the method proceeds to generate an output video sequence. This sequence is refined, with the selection and arrangement of its scenes determined by established viewing quality metrics. Implementing a metric-based approach results in video sequences with low occlusion, and therefore most likely to captivate and engage viewers. By stringing together the most interesting moments, combined with analytical insights gained from the machine learning process, a compelling narrative of the combat sports event is woven.

[0130] In summary, this computer-based approach to automated video production for combat sports events represents a revolutionary change in the field. Utilizing enhanced and intelligent video processing capabilities, it ensures a rich viewer experience, a streamlined production process, and respects the inherent unpredictability and dynamism of combat sports events.

[0131] It can be specified that the at least one input video sequence is multiple input video sequences. In this case, processing steps can be performed on each of the multiple video sequences. The multiple input video sequences can be captured by multiple video cameras.

[0132] Each video sequence can provide a unique view of the event created by its respective viewpoint (e.g., position, angle, and / or zoom factor relative to the combat sports event). Parallel processing of the video sequences provides a more detailed record of the event, leaving fewer blind spots and measuring more nuances of the activities unfolding on the field. Instead of acting on a single video camera source, the object detection algorithm and subsequent method steps of this invention can process each video sequence individually. By doing so, the technique allows for capturing a comprehensive picture of the event from multiple viewpoints, thereby maximizing the value of the output video sequences. This also makes it possible to minimize occlusion in the generated output video sequences.

[0133] In particular, parallel processing of more than one video camera source is advantageous when determining the positioning and movement of fighters. In rapidly evolving events such as combat sports, a single moment can have a significant impact on the audience's understanding and appreciation. Using a single video source, the opportunity to capture such a moment is inherently limited by the given perspective and frame. Furthermore, the use of multiple video cameras contributes to a higher level of fault tolerance. In the event of a technical malfunction, failure, or any other problem with one camera, the system will not lack data to process; the sources from the remaining cameras continue to provide input, thus ensuring uninterrupted execution. Moreover, to improve the stability of the method according to the invention, it is advantageous to use more than one camera, i.e., to transfer the viewpoint of the virtual camera and / or the viewpoint of the generated output video sequence to a video camera with a stable view.

[0134] It can be specified that the method is performed based on multiple machine learning models, wherein performing the method includes: utilizing a sequence of at least two machine learning models, and / or wherein performing the method includes: utilizing at least three machine learning models, wherein the outputs of at least two of the at least three machine learning models are used as inputs to at least one of the at least three machine learning models.

[0135] Including multiple machine learning models in the method according to the invention constitutes a significant enhancement. The method of the invention enables the program to leverage the specialized advantages of various machine learning models, resulting in collaborative, multifaceted operation at runtime. In particular, by using a strategic sequence of machine learning models, a unique combination of the capabilities of each model emerges, which collectively improve the output quality of the method.

[0136] For example, one machine learning model can focus on interpreting an input video sequence to determine regions of interest. This model can be specialized for object detection and tracking, thus ensuring accurate observation of the most critical segments involving the fighters. Another machine learning model can focus on spatial analysis to evaluate the output of the first model, i.e., to determine viewing quality metrics. This model can excel at evaluating occlusion events, strategic angle shifts, camera viewpoints, etc., to deliver weighted quality values.

[0137] One or more of the machine learning models can be implemented as classifiers. For example, an object detection algorithm can classify different regions in a video sequence as regions of interest or not. More sophisticated classifiers can even classify the type of object (i.e., body parts and / or similar objects). Additionally or alternatively, classifiers may also potentially help determine viewing quality metrics, classifying them into categories such as high, medium, or low, or any other suitable level, based on some predefined criteria or thresholds.

[0138] The focus of this concept is on achieving output of generally good quality while simultaneously facilitating real-time operation with low processing resources. Machine learning models can be advantageously selected and ranked based on their individual and combinatorial capabilities. Examples of other combinatorial machine learning models are given in further disclosure below.

[0139] It can be stipulated that at least one region of interest is automatically determined based on a deterministic machine learning model.

[0140] As described above, identifying a combination of regions of interest and viewing quality metrics is particularly advantageous and provides a powerful concept for enhancing the quality of the generated output video sequence.

[0141] Regions of interest—as identified by object detection algorithms—encompass the most critical fighting movements in combat sports. Such examples are seen in boxing matches, where rapid fist movements are particularly relevant. Here, the object detection algorithm might delineate the boxer's fist as the region of interest, effectively encapsulating the segment of the video sequence where the boxer (in this case, the first fighter) is striking. This approach ensures that every significant moment or visually compelling action is not lost in noise but is processed with sufficient attention. Combined with this, a viewing quality metric is provided as a concept for rating regions of interest in a given video sequence based on visibility or clarity. By estimating the viewing quality metric, the system can filter or prioritize the most visible, least occluded action moments in the final output video sequence, and / or select an optimal output video subsequence for a specific action from multiple input video sequences. For example, in a boxing match scene, if the camera angle or the relative positioning of the boxers causes one boxer to obstruct the view of another's striking action, the viewing quality metric associated with that striking sequence will likely be low. Conversely, if the same striking action occurs with minimal or no obstruction in different input video sequences, thus providing clear visibility of the strikes within the video sequence, the viewing quality metric for that sequence will be high. This automatically guides the generation of the output video sequence, ordering it to favor more visually recognizable and relevant fighting actions, thereby maximizing user engagement with and appreciation of combat sports events, as well as the objective quality of the output video sequence.

[0142] Furthermore, it is important to note that, when considering the principles of the invention, the region of interest (ROI) in a video sequence can be a specific subsequence of that video sequence. This subsequence can include portions of video frames or images and can also span temporal phases of the video sequence. Therefore, the ROI is not limited to isolated frames or moments but can be extended to include a set or sequence of frames that collectively contribute to capturing the dynamics and continuity of the fighter's movements. Using machine learning models to determine the ROI and to determine viewing quality metrics is particularly advantageous and can provide enhanced output video sequence quality. In terms of implementation, these two tasks can be incorporated into a single machine learning model. However, providing two separate machine learning models for these two tasks (where the output of, for example, the first machine learning model for determining the ROI serves as the input to, for example, the second machine learning model for determining viewing quality metrics) is particularly advantageous due to processing resources and for facilitating a real-time experience.

[0143] It can be specified that preprocessing of one or more of the at least one video sequence is performed before processing the video sequence, in order to reduce the processing load required to perform the processing. Alternatively or additionally, preprocessing is performed based on a preprocessing machine learning model.

[0144] The concept of performing a preprocessing stage before initiating core video sequence processing introduces several technical advantages. The preprocessing stage primarily prepares and / or refines (one or more) the video sequence for subsequent stages of the method. This can involve activities such as noise filtering, adjusting lighting or contrast, masking and / or padding irrelevant portions of the sequence, introducing labels into the video sequence, marking portions of the video sequence as key combat actions, and / or reducing resolution or frame rate for simplification. These adjustments help reduce redundant and / or irrelevant information, thereby streamlining the input that will be processed by subsequent stages of the method, thus reducing, for example, the demand on processing resources.

[0145] Enriching preprocessing with dedicated machine learning models can further improve the efficiency of this stage. Preprocessing machine learning models can learn to effectively detect and filter redundant and / or irrelevant information, identify relevant parts of video sequences, and optimally calibrate the input video sequences for subsequent tasks.

[0146] A significant advantage of this setup is the reduction in processing load. It alleviates the burden on the remaining stages of the process, thus providing a well-processable foundation for object detection algorithms and other tasks. Furthermore, reduced computational resources and processing time improve overall efficiency. This advantage becomes particularly valuable in real-time and / or live streaming scenarios, where minimizing the time spent from capture to broadcast is crucial. Moreover, the preprocessing step significantly improves the performance of object detection algorithms by providing a clearer, simpler input video sequence, reducing errors, and increasing accuracy.

[0147] It can also be specified that preprocessing includes: splitting video sequences by time period and / or cropping video sequences by image portion.

[0148] Splitting and / or cropping video sequences is highly beneficial. Splitting a video sequence refers to the temporal division of the input data. This involves dividing the entire length of the video sequence into smaller time segments and / or blocks. This can be extremely advantageous when processing long-duration sporting events, simplifying the processing of the video data and optimizing processing time.

[0149] Video sequence cropping can refer to the spatial segmentation of the input data. It's very similar to cropping a photo to focus on a specific object or viewpoint. Figure 1 Similarly, video cropping can involve selecting relevant or interesting parts of a video frame and ignoring the rest.

[0150] This helps to manage computational resources even more effectively. Working on smaller time intervals and focused portions of the image reduces the amount of data to be processed in each iteration step, thus requiring less memory and processing power. Furthermore, segmenting by time and space can lead to a significant reduction in processing time.

[0151] It can be stipulated that the automatic determination of viewing quality metrics is based on the evaluation of machine learning models.

[0152] It can be stipulated that viewing quality indicators be automatically determined taking into account the viewpoint of the corresponding video camera and / or based on the relative positions of the fighter and, optionally, the referee.

[0153] As described above, one of the major challenges that can affect the quality and clarity of video sequences in combat sports is occlusion. This phenomenon can occur in combat sports, particularly when fighters overlap or overlap specific body parts in the video sequence, resulting in obstructed views from certain camera viewpoints. Viewing quality metrics can be determined based on the degree of occlusion and thus taking into account the relative positions of the fighters and, alternatively, the referee. This is particularly advantageous, as described above. The positional relationship can be based on the viewpoint relative to a virtual line drawn between the centers of mass of the two fighters' bodies. If this angle is measured at 90°, the view captures a side-to-side view of the fighters and their movements. Thus, in this case, with minimal occlusion obstructing the view, the visualization and tracking of movement are likely to be generally clear and detailed, resulting in visibility to the viewer. This can correspond to a high value for the viewing quality metric. On the other hand, if this angle is measured at 0°, the view is aligned exactly behind or in front of the fighters. Therefore, one fighter may severely obstruct the view of another, significantly blurring the view of their movements, actions, and expressions. This lack of visibility can correspond to a low value in the viewing quality index.

[0154] It can be specified that generating the output video sequence includes determining the viewpoint of a virtual camera and generating the output video sequence based on that viewpoint. Determining the viewpoint of the virtual camera is preferably performed based on a viewpoint machine learning model, and / or determining the viewpoint of the virtual camera includes one or more of the following: selecting one of the viewpoints of the plurality of video sequences based at least in part on a determined viewing quality metric; performing zoom-in or zoom-out operations, particularly performing zoom-in or zoom-out operations on one or more of the at least one video sequence; performing cropping operations, particularly performing cropping operations on one or more of the at least one video sequence. Thus, for example, the viewpoint of the virtual camera may be a cropped portion of a video sequence captured by a video camera.

[0155] For example, if combat sports were boxing, then identifying the optimal viewpoint would be crucial for effectively capturing the rapid movements and striking exchanges between fighters. This process could be performed using machine learning, which is designed to evaluate the optimal viewpoint based on variables such as the fighters' movements, their position in the ring, available camera angles, and viewing quality metrics, as discussed in more detail above.

[0156] Determining the viewpoint of a virtual camera involves several potential operations. For example, this could involve selecting one viewpoint from multiple video sequences (i.e., video cameras) based on determined viewing quality metrics. This will optimally display the combat action while providing different perspectives on the event. It ensures viewers don't miss important moments during sporting events, significantly enhancing the viewing experience. Furthermore, during combat sports events, this can provide viewers with a more diverse and / or more engaging experience, allowing them to see combat actions from different angles.

[0157] Operations such as zooming in or zooming out can also be performed on one or more segments of a video sequence. For example, in a crucial punch exchange during a boxing match, zooming in would allow viewers to witness the interaction more closely. Conversely, zooming out can provide an overview of the boxer's position and movement, offering additional context. Operations such as cropping can further focus on key events or movements while removing unnecessary elements or environment from the video. This emphasizes the most salient aspects of the sport, thereby increasing viewer engagement and comprehension.

[0158] The method may include: prior to generating an output video sequence, preferably based on the detection of actions such as striking, kicking, and grappling, and more preferably based on a correlation assessment of these actions, determining a proposal for exciting scenes. Additionally or alternatively, the method may include: prior to generating an output video sequence, preferably based on the detection of actions such as striking, kicking, and grappling, and more preferably based on a correlation assessment of these actions, determining combat statistics and / or combat statistics proposals.

[0159] Before generating the output video sequence, the method can be configured to identify highlight proposals as key moments in combat sports events. This aspect of the invention operates by identifying actions such as strikes, kicks, and takedowns and applying algorithms to assess their importance to the overall narrative of the event. For example, a perfectly executed strike or an impressive takedown provides a key moment that can change the outcome of a combat sports event. One example is detecting knockdown sequences in boxing. The technique identifies the movement indicating a powerful punch that leads to a knockdown. It then marks this sequence as highlight proposals and prioritizes them during the creation of the output video sequence. Furthermore, the method can also identify combat statistics or proposals for combat statistics. This step benefits not only viewers but also sports analysts and commentators interested in the quantitative measurement of actions within combat sports. The technique detects actions such as strikes, kicks, and takedowns and uses them to algorithmically determine combat statistics based on the rules and regulations of the respective combat sport. One example of this could involve detecting striking sequences within a boxing match; the invention would track and record the number of strikes thrown by each fighter, the percentage of hits, the frequency and intensity of powerful strikes, and so on. This statistical information provides a numerical representation of the rhythm and intensity of combat sports events and can be used for detailed post-match analysis or even real-time commentary.

[0160] It can be stipulated that the proposed highlights are scored based on user preferences, wherein the selection of highlights is performed to generate the output video sequence. This selection can be performed based on a machine learning model.

[0161] This method can be incorporated into a personalized system that rates highlight scene proposals based on viewer preferences. This allows viewers to customize output video sequences according to their individual interests, enabling them to view extended sequences of their favorite fighters or their preferred moves, such as punches, kicks, or grappling. For example, viewers might prefer scenes characterized by high-intensity clashes or moments where a particular fighter dominates the match. If so, the system will be more favorable in rating scenes that align with these preferences, thus customizing the viewing experience to individual tastes.

[0162] To further facilitate a personalized viewing experience, the method can include selecting highlight scenes for generating an output video sequence. This selection process can be performed based on a machine learning model. The machine learning model can utilize data from several variables—including the user's viewing history, behavior, and explicit preferences—to select the right combination of scenes. For example, if viewers frequently rewatch scenes characterized by successful uppercuts, the machine learning model might prefer highlight scenes with similar actions when generating the output video sequence.

[0163] It can be specified that the output video sequence includes at least a portion of the determined statistics and the proposed highlights, wherein the proposed highlights are included in the overlapping graphics of the output video sequence, particularly during breaks in combat sports events.

[0164] This method facilitates the inclusion of computed statistics in the output video sequence, enhancing the viewing experience with key data of interest. This ensures that viewers can access insightful statistics simultaneously with the real-time progress of the event without switching between different media. Considering a scenario where a boxing match is broadcast as the output video sequence during a live combat sport event, the system could include statistics such as the number of strikes landed by each fighter, the percentage of hits and misses, and the power of the hits. These figures provide viewers with an analytical perspective, allowing them to measure performance in real time. Similar to what has been described above, the selection of statistics can also be based on personalized viewer preferences, particularly through machine learning models.

[0165] According to a second aspect of the invention, there is a provision for using at least two machine learning models to perform the method according to the first aspect of the invention. It may also be specified that the output of at least one machine learning model can be used as the input of at least one different machine learning model.

[0166] According to a third aspect of the present invention, a data processing apparatus, preferably a computer device and / or a server and / or a cloud platform, is provided, including means for performing the method according to the first aspect of the present invention.

[0167] According to a fourth aspect of the invention, the use of a plurality of video cameras in a method according to a first aspect of the invention is provided. The video cameras can be deployed around combat sports events.

[0168] According to a fifth aspect of the invention, a system is provided that includes a plurality of video cameras according to a fourth aspect of the invention and a data processing device according to a third aspect of the invention, wherein the system is configured to perform a method according to a first aspect of the invention.

[0169] According to a sixth aspect of the invention, a computer program or a computer-readable medium having a computer program stored thereon is provided, the computer program including instructions that, when executed by a computer, cause the computer to perform a method according to a first aspect of the invention.

[0170] All features, technical implementation details, and advantages described with respect to any aspect of the invention herein are readily applicable to any other aspect of the invention, with necessary modifications, and vice versa. Attached Figure Description

[0171] The following figures will provide a better understanding of this disclosure: Figure 1 This is a schematic top view of a boxing ring in which the method according to an embodiment of the present invention is performed.

[0172] Figure 2a A first exemplary snapshot of a video frame including a region of interest according to an embodiment of the present invention is shown.

[0173] Figure 2b A second exemplary snapshot of a video frame including a region of interest according to an embodiment of the present invention is shown.

[0174] Figure 3 A first exemplary sequence of video frames showing a cropped region of interest according to an embodiment of the present invention is shown.

[0175] Figure 4 A second exemplary sequence of video frames showing a cropped region of interest according to an embodiment of the present invention is shown.

[0176] Figure 5 An exemplary frame-by-frame signal output for counting hits is shown according to an embodiment of the present invention.

[0177] Figure 6 It is a flowchart including the steps according to an embodiment of the present invention.

[0178] Figure 7 This is a schematic top view of a boxing ring in which the method according to an embodiment of the present invention is performed.

[0179] Figure 8 An exemplary snapshot of the output video sequence according to an embodiment of the present invention is shown.

[0180] Figure 9 Three exemplary snapshots of an input video sequence according to an embodiment of the present invention are shown.

[0181] Figure 10 The illustration shows a series of in-memory frames of a set of exciting moments from a fighting event being prepared according to an embodiment of the present invention.

[0182] Figure 11a This is the first part of a first flowchart that includes the steps of various aspects of the invention, described in different terms.

[0183] Figure 11b This is the second part of a first flowchart that includes the steps of which disclose aspects of the invention in different terms.

[0184] Figure 12This is a second flowchart that includes steps that disclose various aspects of the invention using different wording.

[0185] Figure 13a This is the first part of a third flowchart that includes steps that disclose aspects of the invention in different terms.

[0186] Figure 13b This is the second part of a third flowchart that includes the steps of various aspects of the invention, described in different terms. Detailed Implementation

[0187] Figure 1 A combat sports event 100 is illustrated schematically, specifically a boxing match in a boxing ring, where a first fighter 101 and a second fighter 102 engage in combat. A referee 103 is present in the boxing ring. Around the boxing ring, three video cameras 200 are positioned at different viewpoints to capture a video sequence 201. Specifically, each of the video cameras 200 provides a corresponding video sequence 201. This could, for example, be a real-time continuous video stream.

[0188] In one particular exemplary embodiment of the method for automated video production according to the invention, three video sequences 201 are initially captured by a respective video camera 200. These three video sequences 201 are received by a processing entity, for example, by a data processing device configured to perform the method according to an embodiment of the invention.

[0189] Based on the processing steps, expected statistics for the combat sports event 100 can be generated, preferably continuously throughout the entire duration of the combat sports event 100. The processing steps are described in further detail below.

[0190] Processing the video sequence 201 may include determining at least one region of interest 202 in the video sequence 201 using an object detection algorithm, wherein the at least one region of interest 202 is associated with one of the first fighter 101 or the second fighter 102 of the combat sports event 100, preferably associated with at least one body part of the first fighter 101 or the second fighter 102. As described in detail above, the determination of at least one region of interest 202 can be performed automatically, particularly based on a machine learning model.

[0191] Furthermore, processing the video sequence 201 may include automatically determining activity data of the fighting activities (particularly strikes performed) of the first fighter 101 relative to the second fighter 102 of the fighting sport event 100, based on at least one region of interest 202 in the video sequence 201. As described in detail above, the determination of activity data may be performed by an activity tracking algorithm, particularly a machine learning-based activity tracking algorithm.

[0192] Examples of regions of interest (ROIs) may include the first or second fighter or their body parts, particularly their fists, forearms, elbows, feet, shins, knees, head, torso, shoulders, and / or hips. Furthermore, a ROI may be an image portion equal to the entire video sequence captured by a video camera, and / or may be a virtual camera view constructed from combinations of other ROIs.

[0193] The method steps according to embodiments of the invention are preferably performed by multiple machine learning models (particularly machine learning models that operate and / or are trained individually). In embodiments of the invention, performing the method includes utilizing a sequence of at least two machine learning models. Alternatively or additionally, performing the method according to embodiments of the invention includes utilizing at least three machine learning models, wherein the outputs of at least two of the at least three machine learning models are used as inputs to at least one of the at least three machine learning models, thereby creating a tree structure of machine learning models. Each machine learning model can be adapted to its specific task, i.e., trained to perform its task optimally.

[0194] Alternatively, according to an embodiment of the invention, the method may be performed by a single machine learning model configured to perform processing on video sequence 201 and generate output video sequence 300.

[0195] In embodiments of the invention, the step of automatically determining activity data can be performed without determining the fighter's posture, and preferably without defining and / or detecting key points for the fighter's body parts. Optionally, the automatic determination of activity data can be performed based on image frames (particularly image frame pixels) of at least one region of interest 202 of the video sequence 201.

[0196] Before processing video sequence 201, preprocessing of one or more of the three captured video sequences 201 can be performed to reduce the processing load required for processing. Preferably, preprocessing is performed based on a preprocessing machine learning model, which may be one of the multiple machine learning models mentioned above. In embodiments of the invention, preprocessing may include splitting video sequence 201 by time period and / or cropping video sequence 201 by image portion, particularly according to region of interest. Alternatively or additionally, preprocessing may include filtering noise, adjusting lighting or contrast, masking and / or filling irrelevant portions of video sequence 201, introducing labels into video sequence 201, marking portions of video sequence 201 as portions of main fighting actions, and / or reducing resolution or frame rate for simplification. These adjustments help reduce redundancy and / or irrelevant information, thereby streamlining the input to be processed by subsequent stages of the method, in order to, for example, reduce the demand for processing resources.

[0197] In embodiments of the invention, automatic determination of activity data for combat activities is performed based on a machine learning model. The machine learning model can be one of the multiple machine learning models mentioned above. Furthermore, automatic determination of viewing quality metrics can also be performed to ensure accurate (i.e., based on suitable video data) determination of the activity data. Viewing quality metrics can be determined based on and / or taking into account the viewpoint of the respective video camera 200, and / or on the relative positions of the fighters 101, 102 and optionally the referee 103. Additionally or alternatively, other relative positional aspects discussed in detail above can be considered, such as the relative positions between body parts of the fighters 101, 102 and / or relative positions relative to the referee 103 and / or relative positions relative to the viewpoints of (one or more) video cameras. The viewpoint can be the position of the respective video camera 200 relative to the combat sports event (i.e., the boxing ring in the case of a combat sports event). Furthermore, the viewpoint can include a corresponding angle of view, more preferably a corresponding zoom factor and / or other settings of the respective video camera 200. Therefore, in other words, the viewpoint of the video camera 200 may include the positioning and orientation of the corresponding video camera 200, as well as other optional settings of the corresponding video camera 200.

[0198] In an embodiment of the present invention, the method may include generating statistical data for combat sports events based on activity data.

[0199] In other words, one or more input video sequences 201 can be captured by one or more video cameras 200 statically mounted on a tripod. The input video sequences 201 may alternatively or additionally include data from separate, isolated video camera feeds from television production (i.e., video cameras 200 operated by the cameraman). The input video sequences 201 may alternatively or additionally include only a single dirty source, i.e., a public source streamed to the viewer. The method according to embodiments of the invention can be configured to be unaffected by view transitions, editing, overlay graphics, etc., present in the dirty source.

[0200] In the following text, especially referring to Figures 2a-7 The embodiments of the present invention are explained using different wording. Regarding these different wordings, they can be understood within the context described above, i.e., as species expressions (subordinate conceptual expressions) of the general expressions used above. For example, the following expressions can be understood as follows: the expression "fight" can be understood as a species of the general expression "combat sports event". The expression "camera system" can be understood as a species of the general expression "at least one camera". The expression "camera" can be understood as a genus expression (superordinate conceptual expression) of the species expression "video camera". The expressions "camera video" and "clip" can be understood as species of the general expression "video sequence and / or output video sequence". The expression "fighter" can be understood as a species of the general expression "fighter". The expressions "camera position" and "view" can be understood as species of the general expression "viewpoint". The expression "stats" can be understood as an abbreviation of the expression "statistics".

[0201] Figure 2a and 2b An exemplary snapshot of a video frame including a region of interest according to an embodiment of the present invention is shown. In particular, the region of interest is determined based on the corresponding virtual camera views of the red and blue corners of the boxing ring. Figure 2a It shows left and right separation, while Figure 2b The separation of near and far objects is illustrated. The bounding boxes have different colors, indicating the allocation of the corresponding corners of the ring—which can be red or blue. In other words, a set of target boxes representing regions of interest, i.e., bounding boxes, are shown, which have been created after the generator phase (described below) and are ready to be fed into the classifier phase (described below). For each fighter in this embodiment, there are regions of interest: “full body of the fighter,” “upper body of the fighter,” “head target,” and “body target.” Figure 2a This shows the positions where the left and right position indicators are passed to the classifier. Figure 2b This shows the location where proximity indicators are used and passed to the classifier. The term "classifier" can be understood as a type of machine learning model.

[0202] Figure 3 It shows in Figure 2a The video frame sequence shown is a cropped region of interest virtual camera view around the corresponding frame, including labels (i.e., visual markers) to indicate the target of interest (i.e., region of interest). In this particular embodiment, the visual markers indicate the left-right separation of the fighters 101, 102. Therefore, in other words... Figure 3 against Figure 2a The frames shown before and after illustrate the "full body of the boxer" and "upper body of the boxer" for the red corner, and the "head target" and "body target" for the blue corner. The visual position indicators (i.e., labels) are visual markers. R indicates focus on the object of interest on the right, i.e., fighter 102, and L indicates focus on the object of interest on the left, i.e., fighter 101. The number indicates the frame rate at which the input source is running. These labels help machine learning models(s) adjust their predictions accordingly for different frame rates.

[0203] Figure 4 It shows in Figure 2b The video frame sequence shown is a cropped region of interest virtual camera view around the corresponding frame, including labels (i.e., visual markers) to indicate the target of interest (i.e., region of interest). In this particular embodiment, the visual markers indicate the near-far separation of fighters 101 and 102. Therefore, in other words... Figure 3 against Figure 2b The frames shown before and after illustrate the "full body of the boxer," "upper body of the boxer," and "head target" and "body target" for the red corner, respectively. Visual position indicators (i.e., labels) are visual markers. "C" indicates focusing on a nearby object of interest, i.e., fighter 101, and "F" indicates focusing on a distant object of interest, i.e., fighter 102. The number indicates the frame rate at which the input source is running. These labels help machine learning models(s) adjust their predictions accordingly for different frame rates. This approach allows for reasonable predictions and counts even in cases with severe occlusion and self-occlusion, as shown.

[0204] Figure 5An example of frame-by-frame signal output from a classifier (i.e., a machine learning model from the classifier stage, as described below) is shown, which is then used as input for combat activity determination (i.e., the action activity counting and contact detection stages, as described below). The red corner (left side of the figure) swung 5 blows in a short time span equivalent to 100 frames. The spikes on the attack hit graph indicate the detected blows. Thus, in other words, the figure shows the frame-by-frame signal output for both the red and blue corners for a set of measured variables. Contact detection can be accomplished by matching the target area signal score at the frame index where the hit occlusion was detected. These signal outputs exist independently for each camera and are merged in the fusion and post-processing stages using temporal nonmaximum suppression or similar methods. Only a few metrics are shown here; one can similarly measure defensive techniques such as blocking, parrying, backing away, swaying, diving, rolling, clinching, etc., or defensive postures such as high hold, peek-a-boo defense, basic defense, Philadelphia shell defense, cross defense, long arm defense, etc. Specific target areas can be used to better capture the signals of strikes to the body and head. One can technically use this method directly on the whole body region of interest without any specific target region of interest. This is self-evidently covered by the scope of this disclosure.

[0205] Figure 6 A flowchart is shown, including steps (i.e. stages) according to embodiments of the invention described in different terms. Therefore, Figure 6 Embodiments of the invention are described below using different terminology to support an overall understanding of specific features. Embodiments of the invention may preferably include one or more of the following stages: 1. Detector Phase. This phase can be independent and is identical for each input video frame from the video sequence. Here, a set of regions of interest, as well as the relative positions of each fighter within the video frame, can be detected. For example, left-right, far-near, or top-bottom. This is similar to... Figure 2a and Figure 2b As shown in the image.

[0206] 2. Tracker Stage. Here, the fighters in the corresponding video frames and their associated regions of interest can be assigned identities, i.e., red corner, blue corner, or similar identities.

[0207] 3. Generator Stage. Here, a set of "virtual cameras" can be defined, with bounding boxes defined around the fighter. The virtual cameras can track the fighter's entire body and / or a specific target region, or another region of interest. In each frame, a target region can be created, and then the current region can move toward that target with given horizontal, vertical, and zoom speed parameters. For this, exponential averaging can be used, but any such similar method can be used for a similar effect. Furthermore, the generator stage can add labels (e.g., visual markers) to video frames to indicate which identity should be considered in a given video frame. Visual markers embedded in each video frame can be used, but alternatively, other indicators fed into the neural network can be used to achieve the same effect. For visual markers, see [link to documentation]. Figure 3 and Figure 4 .

[0208] 4. Classifier Stage. In this stage, cropped frame regions from the corresponding regions of interest (ROIs) of various virtual cameras are fed into the action-activity classifier (i.e., the machine learning model). A custom classifier can be used, which operates frame by frame and then saves some information to feed into the classification of the next frame, thus enabling fast processing while remaining accurate because it takes temporal information into account. Alternatively, standard I3D image processing models and / or similar models can be used, similar to a sliding window implementation. However, the processing speed may be lower.

[0209] 5. Counter Phase. In this phase, signals from each classifier are collected and processed to count actions such as swings and hits. For a better understanding, please refer to [link to relevant documentation]. Figure 5 .

[0210] 6. Fusion and Post-processing Stage. In this stage, statistical information from multiple video cameras can be merged using temporal nonmaximum suppression and / or other post-processing methods. The method according to embodiments of the invention can count only what is visible from the view of the respective camera on each camera. Therefore, with multiple cameras, some hits may be classified as “missed” in one view because the target region of interest (i.e., the target area for the corresponding hit) is occluded. However, the same hit may be classified as “hit” (i.e., successful) in another view because the target region of interest (i.e., the target area for the corresponding hit) is visible there to see the hit. Generally, the counting quality will be higher with multiple camera inputs, although the invention can work well and provide reasonable results even using only a single video camera.

[0211] Figure 7A combat sports event 100x is schematically illustrated, specifically a boxing match in a boxing ring, where a first fighter 101x and a second fighter 102x engage in combat. A referee 103x is present in the boxing ring. Around the boxing ring, three video cameras 200x are positioned at different viewpoints to capture video sequences 201x. Specifically, each of the video cameras 200x provides a corresponding video sequence 201x. This could, for example, be a real-time continuous video stream.

[0212] In a particular exemplary embodiment of the method for automated video production according to the present invention, three video sequences 201x are initially captured by a respective video camera 200x. These three video sequences 201x are received by a processing entity, for example, by a data processing device configured to perform the method according to an embodiment of the present invention. Based on the processing steps, a desired output video sequence 300x can be generated. The processing steps are described in further detail below.

[0213] Processing the video sequence 201x may include determining at least one region of interest 202x in the video sequence 201x using an object detection algorithm, wherein the at least one region of interest 202x is associated with one of the first fighter 101x or the second fighter 102x in the combat sports event 100x, preferably associated with at least one body part of the first fighter 101x or the second fighter 102x. As described in detail above, the determination of at least one region of interest 202x can be performed automatically, particularly based on a machine learning model.

[0214] Furthermore, processing the video sequence 201x may include determining a viewing quality metric for at least one region of interest 202x identified in the video sequence 201x, based at least in part on the positional relationship between the first fighter 101x and the second fighter 102x in the combat sports event 100x. As described in detail above, the determination of the viewing quality metric can be performed automatically, particularly based on a machine learning model.

[0215] Examples of regions of interest (ROIs) may include the first or second fighter or their body parts, particularly their fists, forearms, elbows, feet, shins, knees, head, torso, shoulders, and / or hips. Furthermore, a ROI may be an image portion equal to the entire video sequence captured by a video camera, and / or may be a virtual camera view constructed from combinations of other ROIs.

[0216] The method steps according to embodiments of the invention are preferably performed by multiple machine learning models (particularly machine learning models that operate and / or are trained individually). In embodiments of the invention, performing the method includes utilizing a sequence of at least two machine learning models. Alternatively or additionally, performing the method according to embodiments of the invention includes utilizing at least three machine learning models, wherein the outputs of at least two of the at least three machine learning models are used as inputs to at least one of the at least three machine learning models, thereby creating a tree structure of machine learning models. Each machine learning model can be adapted to its specific task, i.e., trained to perform its task optimally.

[0217] Alternatively, according to an embodiment of the invention, the method may be performed by a single machine learning model configured to perform processing on the video sequence 201x and generate an output video sequence 300x.

[0218] Before processing video sequence 201x, preprocessing of one or more of the three captured video sequences 201x can be performed to reduce the processing load required for processing. Preferably, preprocessing is performed based on a preprocessing machine learning model, which may be one of the multiple machine learning models mentioned above. In embodiments of the invention, preprocessing may include: splitting video sequence 201x by time period and / or cropping video sequence 201x by image portion. Alternatively or additionally, preprocessing may include filtering noise, adjusting lighting or contrast, masking and / or filling irrelevant portions of video sequence 201x, introducing labels into video sequence 201x, marking portions of video sequence 201x as portions of main fighting actions, and / or reducing resolution or frame rate for simplification. These adjustments help reduce redundancy and / or irrelevant information, thereby streamlining the input to be processed by subsequent stages of the method, in order to, for example, reduce the demand for processing resources.

[0219] In embodiments of the invention, the automatic determination of viewing quality metrics is performed based on an evaluation machine learning model. The evaluation machine learning model can be one of the multiple machine learning models submitted above. Furthermore, the automatic determination of viewing quality metrics can be performed based on and / or considering the viewpoint of the respective video camera 200x, and / or on the relative positions of the fighters 101x, 102x, and optionally the referee 103x. Additionally or alternatively, other relative positional aspects discussed in detail above can be considered, such as the relative positions between body parts of the fighters 101x, 102x and / or relative positions relative to the referee 103x and / or relative positions relative to the viewpoints of (one or more) video cameras. The viewpoint can be the position of the respective video camera 200x relative to the combat sports event (i.e., the boxing ring in the case of a combat sports event). Furthermore, the viewpoint can include a corresponding angle of view, more preferably a corresponding zoom factor, and / or other settings of the respective video camera 200x. Therefore, in other words, the field of view of the video camera 200x may include the positioning and orientation of the corresponding video camera 200x, as well as other optional settings of the corresponding video camera 200x.

[0220] In embodiments of the invention, the method may include generating an output video sequence 300x, which includes determining a viewpoint of a virtual camera, based on which the output video sequence 300x is generated. Determining the viewpoint of the virtual camera may be performed based on a viewpoint machine learning model, which may be one of the plurality of machine learning models mentioned above. Alternatively or additionally, determining the viewpoint of the virtual camera may include selecting a viewpoint from a plurality of video sequences 201x based at least in part on a determined viewing quality metric. Alternatively or additionally, determining the viewpoint of the virtual camera may include performing zoom-in or zoom-out operations, particularly performing operations on one or more of at least one video sequence 201x. Alternatively or additionally, determining the viewpoint of the virtual camera may include performing cropping operations, particularly performing operations on one or more of at least one video sequence 201x. Alternatively or additionally, determining the viewpoint of the virtual camera may include adjusting the settings of one of the video cameras 200x.

[0221] Regarding the output video sequence 300x generated by embodiments of the present invention, the output video sequence 300x may include highlights of combat sports events and / or statistics of combat sports events. Highlights and / or statistics may be selected, for example, based on individual viewer preferences—preferably set by the viewer—by providing viewer input to a front-end device, which may be particularly capable of displaying the generated output video sequence 300x. Prior to generating the output video sequence 300x, a highlight proposal determination may be performed. This may include detecting actions such as striking, kicking, and grappling. In embodiments of the present invention, the highlight proposal determination may be performed based on a machine learning model, which may be one of the plurality of machine learning models mentioned above. Alternatively or additionally, highlight proposals may be scored based on user preferences, wherein selection of highlights is performed. This may be performed using the machine learning models mentioned above. Additionally or alternatively, generating the output video sequence 300x may include the determined statistics and at least a portion of the proposed highlight scenes. The proposed and selected highlights, particularly during breaks in the combat sports event 100x, can be included as overlay graphics in the output video sequence 300x. The proposed and selected statistical information can, particularly during breaks in the combat sports event 100x, be included as overlay information in the output video sequence 300x.

[0222] exist Figure 8 The example output video scene 300x is illustrated. As can be seen in the figure, a first fighter 201x and a second fighter 202x are engaged in a boxing match. Figure 9 The figure illustrates the three video sequences 201x mentioned above from three video cameras 200x. Video sequence 201x can preferably be preprocessed to add labeling information to facilitate smooth and efficient further processing. Furthermore, as can be seen in the figure, the region of interest 202x is marked with the observed bounding box. Figure 10 The text illustrates a set of proposed and excellent scenarios.

[0223] Figure 11a and Figure 11bA flowchart illustrating one embodiment of the invention, explained in different terms, is shown. First, the process of "recording the match using a camera system with N cameras placed around the boxing ring" is performed. Then, the process of "extracting audio and finding the synchronization time offset between the camera videos" is performed. Next, the process of "splitting the N camera videos" is performed. Then, the process of "detecting the positions of the fighters and the referee" is performed. Then, the process of "quantifying the viewing angle and viewing quality" is performed. Then, the process of "updating the virtual camera positions" is performed. Then, the process of "extracting statistical information and generating highlight proposals" is performed. Then, the process of "selecting active virtual cameras and performing cropping to generate a video stream" is performed. Then, the process of "performing highlight selection and fusion of statistical information" is performed. Then, the process of "adding relevant overlay graphics and inserting highlight clips during round breaks" is performed. Finally, the process of "sending the processed video stream to the user" is performed.

[0224] Figure 12 A flowchart illustrating various aspects of embodiments of the invention, described in different terms, is shown. The flowchart relates to the process of updating a virtual camera viewpoint in an embodiment of the invention. The flowchart includes the step of "acquiring boxer and referee detection." Furthermore, the flowchart includes the step of "calculating target view coordinates." Additionally, the flowchart includes the step of "updating the current virtual camera coordinates." Furthermore, the flowchart includes the step of "cropping and extracting the virtual camera view."

[0225] Figure 13a and Figure 13b The flowcharts include various parts describing embodiments of the invention in different terms, particularly including the real-time generation of exciting scenes. For clarity, refer to the accompanying drawings regarding the content of the flowcharts, as they contain several branches that cannot be concisely represented in text.

[0226] about Figure 11a , Figure 11b , Figure 12 , Figure 13a and Figure 13bThe following expressions, depending on their wording, can be understood as follows: The expression "competition" can be understood as a category of the general expression "combat sports event". The expression "camera system" can be understood as a category of the general expression "at least one video camera". The expression "camera" can be understood as a genus expression of the category expression "video camera". The expressions "camera video" and "editing" can be understood as categories of the general expression "video sequence and / or output video sequence". The expression "boxer" can be understood as a category of the general expression "fighter". The expressions "viewing angle" and "viewing quality" can be understood as genus expressions within the context of the general expression "viewing quality indicators" described above. The expressions "camera position" and "view" can be understood as categories of the general expression "viewpoint". The expression "statistics" can be understood as an abbreviation of the expression "statistics". The expression "video stream" can be understood as a category of the general expression "output video sequence". The expression "target view" can be understood as a category of the general expression "viewpoint of video camera and / or virtual camera".

[0227] Other implementation schemes are disclosed below: Implementation Scheme 1. A computer-implemented method for automatic video production of combat sports events (100x), the method comprising: Receive at least one input video sequence (201x) of a combat sports event (100x) captured by at least one video camera (200x). Processing the at least one video sequence includes: Using object detection algorithms, particularly machine learning-based object detection algorithms, at least one region of interest (202x) in the video sequence (201x) is automatically determined, wherein the at least one region of interest (202x) is associated with the first fighter (101x) of the combat sports event (100x), preferably with at least one body part of the first fighter (101x); and Based at least in part on the positional relationship between the first fighter (101x) and the second fighter (102x) in the combat sports event (100x), the viewing quality index of the at least one region of interest (202x) identified in the video sequence (201x) is automatically determined; and The output video sequence (300x) is generated based at least in part on the determined viewing quality metric.

[0228] Implementation Scheme 2. The computer-implemented method according to Implementation Scheme 1, wherein the at least one input video sequence (201x) is a plurality of input video sequences, and wherein processing is performed on each of the plurality of video sequences (201x), and wherein preferably, the plurality of input video sequences are captured by a plurality of video cameras (200x).

[0229] Implementation Scheme 3. A computer-implemented method according to any one of the foregoing Implementation Schemes 1-2, wherein the method is performed based on multiple machine learning models, wherein Performing the method includes: utilizing a sequence of at least two machine learning models, and / or The method includes: utilizing at least three machine learning models, wherein the outputs of at least two of the at least three machine learning models are used as the input of at least one of the at least three machine learning models.

[0230] Implementation Scheme 4. A computer-implemented method according to any one of the preceding Implementation Schemes 1-3, wherein automatic determination of at least one region of interest (202x) is performed based on a determined machine learning model.

[0231] Implementation Scheme 5. A computer-implemented method according to any one of the foregoing embodiments 1-4, wherein, prior to processing the at least one video sequence (201x), preprocessing of one or more of the at least one video sequence (201x) is performed to reduce the processing load required to perform the processing, wherein the preprocessing is preferably performed based on a preprocessing machine learning model.

[0232] Implementation Scheme 6. The computer-implemented method according to Implementation Scheme 5, wherein the preprocessing includes: splitting the at least one video sequence (201x) according to time periods and / or cropping the at least one video sequence (201x) according to image portions.

[0233] Implementation Scheme 7. The computer-implemented method according to any one of the preceding Implementation Schemes 1-6, wherein the automatic determination of the viewing quality index is performed based on an evaluation machine learning model.

[0234] Implementation Scheme 8. A computer-implemented method according to any one of the preceding Implementation Schemes 1-7, wherein an automatic determination of viewing quality indicators is performed, taking into account the viewpoint of the respective video camera (200x) and / or based on the relative positions of the fighter and an optional referee (103x).

[0235] Implementation Scheme 9. A computer-implemented method according to any one of Implementation Schemes 1-8 above, wherein generating the output video sequence (300x) comprises: determining the viewpoint of a virtual camera, the output video sequence (300x) being generated based on the viewpoint, wherein determining the viewpoint of the virtual camera is preferably performed based on a viewpoint machine learning model, and / or wherein determining the viewpoint of the virtual camera includes one or more of the following: One of the viewpoints from the plurality of video sequences (201x) is selected, at least in part, based on the determined viewing quality metric. Perform zoom-in or zoom-out operations, particularly performing zoom-in or zoom-out operations on one or more of the at least one video sequence (201x). Perform a cropping operation, specifically performing a cropping operation on one or more of the at least one video sequence (201x).

[0236] Implementation Scheme 10. The computer-implemented method according to any one of the foregoing Implementation Schemes 1-9 further includes: Before generating the output video sequence (300x), preferably based on the detection of actions such as hitting, kicking and / or slamming, and more preferably based on a correlation assessment of them, exciting scene proposals are determined.

[0237] Implementation Scheme 11. The computer-implemented method according to Implementation Scheme 10, wherein the highlight scene proposal is rated based on user preferences, wherein the selection of highlight scenes is performed to generate the output video sequence (300x).

[0238] Implementation Scheme 12. The computer-implemented method according to Implementation Scheme 10 or 11, wherein the output video sequence (300x) includes at least a portion of the proposed highlights and optionally determined statistics of the combat sports event, wherein the proposed highlights are included as overlay graphics in the output video sequence (300x) particularly during the breaks of the combat sports event (100x).

[0239] Implementation Scheme 13. At least two machine learning models are used to perform the method according to any one of Implementation Schemes 1 to 12, preferably wherein the output of at least one machine learning model is used as the input of at least one other machine learning model.

[0240] Implementation Scheme 14. A data processing apparatus, preferably a computer device and / or a server and / or a cloud platform, including means for performing the method according to any one of Implementation Schemes 1 to 12.

[0241] Implementation Scheme 15. A computer program or a computer-readable medium having a computer program stored thereon, the computer program including instructions that, when executed by a computer, cause the computer to perform the method according to any one of Implementation Schemes 1 to 12.

[0242] Some embodiments of the present invention may be based on the use of machine learning models or machine learning algorithms. Machine learning can refer to algorithms and statistical models that allow computer systems to perform specific tasks without using explicit instructions but rely on models and inferences. For example, in machine learning, instead of rule-based data transformations, data transformations inferred from the analysis of historical and / or training data can be used. For example, machine learning models or machine learning algorithms can be used to analyze the content of images. To enable a machine learning model to analyze the content of an image, the machine learning model can be trained using training images as input and training content information as output. By training the machine learning model with a large number of training images and / or training sequences (e.g., words or sentences) and associated training content information (e.g., labels or annotations), the machine learning model "learns" to recognize the content of images, making it possible to use the machine learning model to recognize the content of images not included in the training data. The same principle can also be used for other kinds of sensor data: by training a machine learning model using training sensor data and a desired output, the machine learning model "learns" the transformation between sensor data and output, which can be used to provide outputs based on non-training sensor data provided to the machine learning model. The provided data (e.g., sensor data, metadata, and / or image data) can be preprocessed to obtain feature vectors, which are used as input to a machine learning model.

[0243] Machine learning models can be trained using training input data. The example specified above uses a training method known as "supervised learning." In supervised learning, a machine learning model is trained using multiple training samples, where each sample can include multiple input data values ​​and multiple expected output values; that is, each training sample is associated with an expected output value. By specifying both the training samples and the expected output values, the machine learning model "learns" which output value to provide based on input samples similar to those provided during training. In addition to supervised learning, semi-supervised learning can also be used. In semi-supervised learning, some training samples lack corresponding expected output values. Supervised learning can be based on supervised learning algorithms (e.g., classification algorithms, regression algorithms, or similarity learning algorithms). Classification algorithms can be used when the output is restricted to a finite set of values ​​(categorical variables), i.e., when the input is classified into one of a finite set of values. Regression algorithms can be used when the output can have any value (within a certain range). Similarity learning algorithms can be similar to both classification and regression algorithms, but are based on learning from examples using a similarity function that measures how similar or related two targets are. In addition to supervised or semi-supervised learning, unsupervised learning can also be used to train machine learning models. In unsupervised learning, input data may be provided (only), and unsupervised learning algorithms can be used to find structure within the input data (e.g., finding commonalities in the data by grouping or clustering the input data). Clustering involves assigning input data, which includes multiple input values, into subsets (clusters) such that input values ​​within the same cluster are similar according to one or more (predefined) similarity criteria, while being dissimilar to input values ​​included in other clusters. Reinforcement learning is a third group of machine learning algorithms that can be used to train machine learning models. In reinforcement learning, one or more software actors (called "software agents") are trained to take actions in an environment. Rewards are calculated based on the actions taken. Reinforcement learning is based on training the one or more software agents to choose actions that increase the cumulative reward, thereby causing the software agents to improve on the tasks they are given (as demonstrated by the increased rewards).

[0244] Furthermore, certain techniques can be applied to certain machine learning algorithms. For example, feature learning can be used. In other words, machine learning models can be trained at least partially using feature learning, and / or machine learning algorithms can include feature learning components. Feature learning algorithms (which may be called representation learning algorithms) can retain information from their inputs but also transform them in a way that makes them useful, often as a preprocessing step before performing classification or prediction. For example, feature learning can be based on principal component analysis or cluster analysis.

[0245] In some embodiments, anomaly detection (i.e., outlier detection) can be used, which is designed to provide the identification of input values ​​that are suspicious due to their significant difference from most of the input or training data. In other words, anomaly detection can be used to at least partially train a machine learning model, and / or the machine learning algorithm may include anomaly detection components.

[0246] In some embodiments, machine learning algorithms can use decision trees as predictive models. In other words, machine learning models can be based on decision trees. In a decision tree, observations about an item (e.g., a set of input values) can be represented by branches of the decision tree, and the output value corresponding to that item can be represented by the leaves of the decision tree. Decision trees can support both discrete and continuous values ​​as output values. If discrete values ​​are used, the decision tree can be called a classification tree; if continuous values ​​are used, the decision tree can be called a regression tree.

[0247] Association rules are another technique that can be used in machine learning algorithms. In other words, a machine learning model can be based on one or more association rules. Association rules are created by identifying relationships between variables in a large amount of data. Machine learning algorithms can identify and / or utilize one or more relationship rules, which represent knowledge derived from the data. Rules can be used, for example, to store, manipulate, or apply knowledge.

[0248] Machine learning algorithms are typically based on machine learning models. In other words, the term "machine learning algorithm" can refer to a set of instructions that can be used to create, train, or use a machine learning model. The term "machine learning model" can refer to a data structure and / or a set of rules that represent the learned knowledge (e.g., training performed based on a machine learning algorithm). In implementations, the use of a machine learning algorithm can mean the use of a base machine learning model (or multiple base machine learning models). The use of a machine learning model can mean that the machine learning model and / or the data structures / rule set constituting the machine learning model were trained by a machine learning algorithm.

[0249] For example, a machine learning model can be an artificial neural network (ANN). An ANN is a system inspired by biological neural networks (such as those found in the retina or brain). An ANN consists of multiple interconnected nodes and multiple connections between the nodes, called edges. There are typically three types of nodes: input nodes that receive input values, hidden nodes that are connected to other nodes (only) and output nodes that provide output values. Each node can represent an artificial neuron. Each edge can transfer information from one node to another. The output of a node can be defined as a (non-linear) function of its inputs (e.g., the sum of its inputs). The node's input can be used in the function based on the "weights" of the edges or nodes that provide that input. The weights of nodes and / or edges can be adjusted during the learning process. In other words, training an ANN can involve adjusting the weights of the nodes and / or edges of the ANN to achieve the desired output for a given input.

[0250] Alternatively, the machine learning model can be a Support Vector Machine (SVM), a Random Forest model, or a Gradient Boosting model. A Support Vector Machine (SVM), also known as a Support Vector Network, is a supervised learning model with an associated learning algorithm that can be used to analyze data (e.g., in classification or regression analysis). An SVM can be trained by providing inputs with multiple training input values ​​belonging to one of two categories. The SVM can then be trained to assign new input values ​​to one of those two categories. Alternatively, the machine learning model can be a Bayesian Network, which is a probabilistic directed acyclic graph (DAG) model. A Bayesian Network can use a DAG to represent a set of random variables and their conditional dependencies. Alternatively, the machine learning model can be based on a genetic algorithm, which is a search algorithm and heuristic technique that simulates the process of natural selection.

[0251] Although some aspects have been described in the context of the apparatus, it is clear that these aspects also represent descriptions of the corresponding methods, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent descriptions of items or features of the corresponding block or apparatus.

[0252] Some or all of the method steps can be performed by (or using) a hardware device, such as a processor, microprocessor, programmable computer, or electronic circuit. Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or software. This implementation can be performed using a non-transitory storage medium, such as a digital storage medium (e.g., floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory), having electronically readable control signals stored thereon that cooperate with (or are capable of cooperating with) a programmable computer system to perform the corresponding methods. Therefore, the digital storage medium can be computer-readable.

[0253] Some embodiments of the present invention provide a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform one of the methods described herein.

[0254] Typically, embodiments of the present invention can be implemented as a computer program (product) having program code that, when run on a computer, performs one of the methods. This program code may, for example, be stored on a machine-readable medium. Other embodiments include a computer program for performing one of the methods described herein, stored on a machine-readable medium. In other words, an embodiment of the present invention is therefore a computer program having program code for performing one of the methods described herein when run on a computer.

[0255] Another embodiment of the invention provides a storage medium (or data carrier, or computer-readable medium) including a computer program stored thereon for performing one of the methods described herein when the computer program is executed by a processor. Data carriers, digital storage media, or recording media are typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a storage medium.

[0256] Another embodiment of the invention provides a data stream or signal sequence representing a computer program for performing one of the methods described herein. This data stream or signal sequence may, for example, be configured to be transmitted via a data communication connection (e.g., via the Internet).

[0257] Another embodiment of the invention provides a processing apparatus, such as a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

[0258] Another embodiment of the present invention provides a computer having a computer program installed thereon for performing one of the methods described herein.

[0259] Another embodiment of the invention provides an apparatus or system configured to transmit (e.g., electronically or optically) a computer program to a receiver for performing one of the methods described herein. The receiver may be, for example, a computer, mobile device, memory device, etc. The apparatus or system may include, for example, a file server for transmitting the computer program to the receiver.

[0260] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0261] Reference symbol 100 Fighting Sports Events 101 First Fighter 102 Second Fighter 103 referee 200 (or more) video cameras 201 (one or more) video sequences 202 (one or more) regions of interest 100x Fighting Sports Events 101x First Fighter 102x Second Fighter 103xReferee 200x (one or more) video cameras 201x (one or more) video sequences 202x (one or more) regions of interest 300x (one or more) output video sequences

Claims

1. A computer-implemented method for automating the evaluation of combat sports events (100), the method comprising: Receive at least one input video sequence (201) of a combat sports event (100) captured by at least one video camera (200). Processing the at least one video sequence (201) includes: At least one region of interest (202) in the video sequence (201) is automatically determined using a target detection algorithm, particularly a machine learning-based target detection algorithm, wherein the at least one region of interest (202) is associated with the first fighter (101) of the combat sports event (100), preferably with at least one body part of the first fighter (101); and Based on the at least one region of interest (202) in the video sequence (201), activity data of the fighting activities of the first fighter (101) relative to the second fighter (102) in the fighting sport event (100), particularly the activity data of the strikes performed, is automatically determined, wherein the determination of the activity data is performed by an activity tracking algorithm, particularly by a machine learning-based activity tracking algorithm; and Statistical data of the combat sports event (100) are generated based on the activity data, preferably continuously generated throughout the entire duration of the combat sports event (100).

2. The computer-implemented method according to claim 1, wherein, Automatic determination of activity data is performed without determining the posture of the fighters (101, 102) and preferably without defining and / or detecting key points for the body parts of the fighters (101, 102), wherein optionally, the automatic determination of activity data is performed based on image frames, in particular image frame pixels, of the at least one region of interest (202) of the video sequence (201).

3. The computer-implemented method according to any one of the preceding claims, wherein the method is performed based on multiple machine learning models, wherein Performing the method includes: Utilizing sequences of at least two machine learning models, and / or The method includes: utilizing at least three machine learning models, wherein the outputs of at least two of the at least three machine learning models are used as the input of at least one of the at least three machine learning models.

4. The computer-implemented method according to any one of the preceding claims, wherein, Before processing the at least one video sequence (201), preprocessing is performed on one or more of the at least one video sequence (201) to reduce the processing load required to perform the processing, wherein the preprocessing is preferably performed based on a preprocessing machine learning model.

5. The computer-implemented method according to claim 4, wherein, The preprocessing includes splitting the at least one video sequence according to time periods (201) and / or cropping the at least one video sequence according to image portions (201).

6. The computer-implemented method according to any one of claims 4-5, wherein, The preprocessing includes: associating at least one tag with the at least one region of interest (202) and / or with the video sequence (201), wherein the at least one tag helps to associate the region of interest (202) with the first fighter (101), and / or wherein the at least one tag provides additional information that enables the determination of activity data, particularly statistical data, based on the additional information.

7. The computer-implemented method according to any one of the preceding claims, wherein, The at least one region of interest (202) in the video sequence (201) is a subsequence of the video sequence (201) that includes a time period and / or a cropped portion of the image of the video sequence (201), wherein preferably, the at least one region of interest (202) is marked by a bounding box representing a corresponding image frame portion of the video sequence (201).

8. The computer-implemented method according to any one of the preceding claims, wherein, Each of the at least one region of interest (202) is associated with a fighter, that is, with the first fighter (101) or the second fighter (102); and Optionally, if the corresponding region of interest (202) is associated with the first fighter (101), then only the activity data of the fighting activities of the first fighter (101) is determined, that is, the activity data of the fighting activities of the second fighter (102) is ignored.

9. The computer-implemented method according to any one of the preceding claims, wherein, Based on at least one feature included in the video sequence (201), the at least one region of interest (202) is tracked within the video sequence (201).

10. The computer-implemented method according to any one of the preceding claims, wherein, The determination of activity data and / or generation of statistics is performed, which includes: executing a dual activity avoidance algorithm to avoid dual activity tracking, wherein preferably, the dual activity avoidance algorithm is based on nonmaximum suppression.

11. The computer-implemented method according to any one of the preceding claims, wherein, The processing also includes: automatically determining a viewing quality index of the at least one region of interest (202) identified in the video sequence (201) based at least in part on the positional relationship between the first fighter (101) and the second fighter (102) of the combat sports event (100) and / or the viewpoint of the corresponding video camera (200), wherein the determination of the viewing quality index is preferably performed based on a viewing evaluation machine learning model.

12. The computer-implemented method according to claim 11, wherein, If processing is performed on multiple video sequences (201) that display the same specific fighting activity in at least one corresponding region of interest (202), the selection of at least one reliable video sequence (201) including the corresponding region of interest (202) displaying the specific fighting activity is made based on the viewing quality metric, wherein the automatic determination of activity data is performed based on the selection.

13. The use of at least two machine learning models to perform the method according to any one of claims 1 to 12, preferably wherein the output of at least one machine learning model is used as the input of at least one other machine learning model.

14. A data processing apparatus, preferably a computer device and / or a server, comprising means for performing the method according to any one of claims 1 to 12.

15. A computer program or a computer-readable medium having a computer program stored thereon, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 12.

Images