Video assistant referee for chess

Abstract

The present technical solution relates to systems for automating chess tournaments, which are intended to digitize games, record violations and analyze play. What is claimed is a method for implementing a video assistant referee and broadcasting games using a computer device comprising a processor, a random-access memory and executable instructions for performing the following steps: processing a video stream from a camera, identifying a chessboard and creating a virtual model thereof; tracking the movement of pieces, while synchronizing the virtual field of play with the real field of play; checking the validity of moves according to the rules of chess; activating predefined triggers upon detection of specific playing situations; and transmitting the current state of a virtual board to a server for broadcasting and digitizing moves.

Description

[0001] VIDEO ASSISTANT CHESS ARBITER

[0002] Field of technology

[0003] This technical solution relates to solutions used in the organization of chess tournaments for the digitalization of chess games, video recording of violations in chess, and game analysis.

[0004] State of the art

[0005] In recent years, video technology has made it possible to view footage captured by one or more cameras in real time and perform certain actions on these videos in sports. The incidence of foul play and violations in sporting events has grown exponentially in recent years. Various solutions have been developed that allow sports referees to monitor the entire process of a sporting event.

[0006] The use of such technologies allows for improved refereeing at sporting events, such as chess competitions. Video assistant referees (VARs) take chess competitions and the refereeing process to a whole new level. In the context of chess competitions, recorded data provides a valid means of confirming results and resolving disputes in tournaments.

[0007] The presented invention relates to a system of embedded arbitration assist devices applicable to chess, which can be integrated into elements designed to assist in making more accurate decisions in controversial situations using machine learning. A video assistant referee (VAR) in chess records and analyzes chess moves in real time, which contributes to increased fairness, a reduction in refereeing errors, and more accurate refereeing decisions.

[0008] A solution is known (CN114797079, published on July 29, 2022), which discloses a method and device for identifying chess pieces and automatically placing chess pieces, ensuring stable movement of a mechanical arm during the automatic placement of chess pieces, the accuracy of placing chess pieces and recognizing chess pieces.

[0009] The disadvantage of this solution is the lack of ability to identify violations in chess games.

[0010] The closest to the claimed invention is a method for detecting a chess surface and correcting errors and a device for a chess-playing robot (CN114821306, published 06.04.2022), which ensures the determination of whether the chess pieces are moving in a given period of time and the generation of hints for correcting errors for the result of the chess movement in a target period of time.

[0011] The disadvantage of the known solution is the lack of the ability to provide automatic support for improved refereeing at sporting events in the event of controversial situations, as well as the lack of the ability to apply this solution at sporting competitions.

[0012] The technical result is an increase in the efficiency and accuracy of recognizing pieces on a chessboard and an increase in the efficiency of game analysis (including a reduction in the number of errors), an increase in the efficiency of the video assistant arbiter and the broadcast of chess tournaments, and an increase in the efficiency of using the resources of a computer device by implementing the methods described below.

[0013] In one embodiment, a method for implementing a video assistant to a chess arbiter and broadcasting chess games comprises the following steps: receiving a video stream; determining the active chessboard in the video stream; recognizing the active chessboard in the video stream and generating a virtual playing field corresponding to its state; analyzing the actions of the players with the pieces on the chessboard in the video stream and tracking the chess pieces on the chessboard, updating the state of the virtual playing field with each move of the players; analyzing the arrangement of the chess pieces on the virtual playing field taking into account the current and previous actions of the players and their compliance with the rules of chess; executing the appropriate trigger in response to the detection of predetermined chess situations;transmit the current state of the virtual playing field to a remote server to carry out broadcasting / digitization (translation of players' moves, game data into digital form, using the selected notation / coding system) of the players' moves.

[0014] In one embodiment, a method for implementing a video assistant for a chess arbiter includes the following steps: receiving a video stream; determining an active chessboard in the video stream; recognizing the active chessboard in the video stream and generating a virtual playing field corresponding to its state; recognizing a chess clock in the video stream; analyzing the actions of players with pieces on the chessboard in the video stream and tracking the chess pieces on the chessboard, updating the state of the virtual playing field at each move of the players, with reference to the recognized chess clock; analyzing the arrangement of chess pieces on the virtual playing field, taking into account the actions of the players and their compliance with the rules of chess; executing an appropriate trigger in response to the detection of predetermined chess situations; transmitting the current state of the virtual playing field to a remote server.

[0015] In some embodiments, a method for implementing a video assistant to a chess arbiter, using a mobile / computer device including at least (optionally a camera), a processor, RAM and machine-readable instructions, includes the following steps: receiving a video stream; determining an active chessboard in the video stream; recognizing the active chessboard in the video stream and forming a virtual playing field corresponding to its state; recognizing a chess clock in the video stream; recognizing the actions of players with chess pieces on the chessboard in the video stream; in response to recognizing the action of a player with pieces on the chessboard: tracking the chess pieces on the chessboard, updating the state of the virtual playing field at each move of the players, with reference to the recognized chess clock; analyzing the arrangement of the chess pieces on the virtual playing field, taking into account the action of the player and their compliance with the rules of chess;in response to identifying predetermined chess situations, executing a corresponding trigger; transmitting the current state of the virtual playing field to a remote server. In some embodiments, the method for implementing a video assistant to a chess arbiter and broadcasting chess games includes the following steps: receiving a video stream; determining an active chessboard in the video stream; recognizing the active chessboard in the video stream and generating a virtual playing field corresponding to its state; recognizing a chess clock in the video stream; recognizing the actions of players with chess pieces on the chessboard in the video stream; in response to recognizing an action of a player with pieces on the chessboard: i. tracking the chess pieces on the chessboard, updating the state of the virtual playing field with reference to the recognized chess clock ii. analyzing the arrangement of the chess pieces on the virtual playing field taking into account the action of the player and their compliance with the rules of chess;iii. in response to the detection of pre-defined chess situations, the corresponding trigger is executed; the current state of the virtual playing field is transmitted to a remote server for broadcasting / digitizing the players' moves.

[0016] In one embodiment, a method for recognizing and analyzing chess moves using a mobile device comprising at least a camera, a processor, RAM, and machine-readable instructions comprising the following steps: obtaining at least one video frame; determining the coordinates of the corners of chessboards on the obtained video frame and determining the active chessboard; determining a fragment of the video frame corresponding to the active chessboard; forming a multi-channel matrix of representing the playing field and the pieces located on it by processing the fragment of the video frame of the active chessboard obtained in the previous step with a pool of models; analyzing the arrangement of the chess pieces on the playing field taking into account the rules of chess; in response to identifying one or a combination of predetermined chess situations, performing a corresponding indication.

[0017] In some implementations, a predetermined chess situation may be one or a combination of the following situations: checkmate, stalemate, draw by the threefold repetition rule, draw due to insufficient pieces, draw due to exhaustion of the move limit, illegal moves, en passant capture, castling, touch-move, release hand-move.

[0018] In some embodiments, to determine the coordinates of the corners of chessboards in a video frame, the chessboards are determined, closely located chessboards are separated, and contours of the chessboards are obtained, each of which is approximated by at least one rectangle, after which the NMS method is applied and a weighted averaging of the coordinates of the closely located corner points is performed.

[0019] In some embodiments, chessboards are determined in the obtained video frame by the following methods: the size of the obtained video frame is reduced, bringing it to a predetermined size; a board mask, a board edge mask, and board corner points are obtained by processing the video frame reduced in the previous step using a convolutional neural network; In some embodiments, to separate closely spaced chessboards, board edge masks are subtracted from the board mask, the obtained image is binarized according to a predetermined threshold, after which the obtained contours are approximated by quadrangles.

[0020] In some embodiments, the multi-channel gameboard matrix is ​​a nine-channel 8x8 matrix, where 8x8 is the representation of the gameboards, four channels provide the representation of four classes of the board: empty board, white piece, black piece, hand blocking, and the remaining five channels provide the representation of the type of the piece: pawn, knight, bishop, rook, king / queen.A method for recognizing and analyzing chess moves using a mobile device comprising at least a camera, a processor, random access memory, and machine-readable instructions comprising the following steps: receiving a video stream; dividing the video stream into video frames; determining the coordinates of the corners of the chessboards on each video frame and determining the active chessboard; determining for each video frame a fragment corresponding to the active chessboard; forming for each video frame a multi-channel matrix of representation of the playing field and the pieces located thereon by processing the fragment of the video frame of the active chessboard obtained in the previous step with a model or a pool of models; in response to a change in the multi-channel matrix of representation of the playing field relative to the previous video frame, analyzing the arrangement of the chess pieces on the playing field taking into account the rules of chess; in response to the detection of predetermined chess situations, performing the corresponding indication.

[0021] In some embodiments, the trigger comprises a predetermined indication and a predetermined action or set of actions.

[0022] In some implementation variants, a predetermined chess situation may be one of the following: checkmate, stalemate, draw, draw by the threefold repetition rule, draw due to insufficient pieces, draw due to exhaustion of the move limit, illegal moves, "en passant capture", castling, "touch-move", "release hand - move made".

[0023] In some embodiments, the detection and recognition of a chess clock in a video stream is additionally carried out, after which the actions of the players are linked to the chess clock.

[0024] In some embodiments, the remote server is configured to determine the best moves in a game and display and broadcast them.

[0025] In some embodiments, the indication of the best moves is carried out by displaying a video fragment of the best moves with added video / audio effects.

[0026] A method for implementing a video assistant to a chess arbiter and broadcasting chess games using a computer device that includes at least a processor, RAM, and machine-readable instructions comprises the following steps: receiving (using the processor) a video stream (from the device's camera); determining and recognizing an active chessboard in the video stream and forming a virtual playing field corresponding to its state (using the processor); analyzing the actions of the players with pieces on the chessboard in the video stream and tracking the chess pieces on the chessboard, updating the state of the virtual playing field with each move of the players (using the processor); analyzing the arrangement of the chess pieces on the virtual playing field taking into account the current and previous moves of the players and their compliance with the rules of chess (using the processor);in response to the identification of pre-defined chess situations as a result of the analysis, the corresponding trigger is executed (using the processor); the current state of the virtual playing field is transmitted to a remote server (using the processor) for the purpose of broadcasting and / or digitizing the players' moves.

[0027] Brief description of the figures

[0028] Fig. 1 shows an illustrative pipeline of operation of some aspects of the technical solution.

[0029] Fig. 2 shows the sequence of steps of the method for recognizing and analyzing chess moves (a particular embodiment).

[0030] Fig. 3 shows the sequence of steps of the method for implementing a video assistant for a chess arbiter and broadcasting chess games (a specific embodiment).

[0031] Fig. 4 shows a particular embodiment of the method according to Fig. 3.

[0032] Fig. 5 shows an illustrative example of displaying a chess broadcast on the viewer’s device.

[0033] Fig. 6 shows an illustrative example of displaying a chess tournament for several games at the viewer’s device.

[0034] Detailed description

[0035] Below are some terms and definitions necessary for understanding the essence of the technical solution.

[0036] The chessboard includes an 8x8 playing field.

[0037] U-Net is a convolutional neural network. Its architecture is a fully connected convolutional network modified to work with a smaller number of examples (training images) and perform more accurate segmentation.

[0038] Binarization is the conversion of a color (or grayscale) image into a two-color black-and-white image. The key parameter of this conversion is the threshold t—the value with which the brightness of each pixel is compared. Based on the comparison results, each pixel is assigned a value of 0 or 1.

[0039] A bounding box is a set of coordinates that define a specific area of ​​an image, most often in the form of a rectangle.

[0040] Dropout is a regularization technique used in neural network training. It involves randomly dropping (disabling) a certain percentage (e.g., a predetermined value of 30%) of neurons in the network at each training stage. This prevents the network from relying on any one group of neurons and forces it to learn more robust features distributed throughout the network, thus preventing overfitting. Regularization is a technique used to prevent a model from overfitting the training data, which typically occurs by increasing variance. The idea behind regularization is to introduce additional constraints or penalties on the size and / or complexity of the model.

[0041] A trigger is a predefined procedure or function executed when a certain event and / or condition occurs. Triggers can be defined as scripts or other methods that allow for the execution of actions or operations.

[0042] From here on, when it is implied that some parameters can be set, it should be taken into account that in some cases such parameters can be set by the developer, the system administrator, or some privileged user of the system, and in some cases by the end user (for example, a judge / referee, the user of the device broadcasting the tournament).

[0043] A method for recognizing and analyzing chess moves using a mobile device that includes at least a camera, a processor, RAM, and machine-readable instructions, comprising the following steps (Fig. 1, Fig. 2):

[0044] Receive at least one video frame

[0045] In some implementations, a video frame (video data, image) is obtained using a personal mobile device with a video / photo camera, such as a smartphone, tablet, or laptop. In some implementations, a desktop PC with an external video / photo camera may be used. In some implementations, the device receives a video stream from the device's own camera.

[0046] In some implementations, a video stream (streaming video) is received and divided into individual video frames. Video frames can be selected either at a fixed rate, such as every Nth frame (where N > 1), or dynamically, depending on the performance of the device implementing the method. For example, when the device is first launched, it can be tested to determine its current performance, indicating how many frames per second (fps) the device can process.

[0047] In some implementations, a camera may be used, with frames taken at a certain frequency, for example, several frames per second.

[0048] The coordinates of the corners of the chessboards are determined on the received video frame and the active chessboard is determined

[0049] In one embodiment, the corners of the board are determined as follows:

[0050] • Reduce the size of the received video frame, bringing it to a pre-set size;

[0051] • A board mask, a board edge mask, and board corner points are obtained by processing the video frame (100) reduced in the previous step using a convolutional neural network (110);

[0052] • Separate closely spaced chessboards.

[0053] In some implementations, the resulting video frame is reduced (resized) to 96x176 pixels.

[0054] In some implementations, a UNet-like convolutional neural network (CNN) is used as a convolutional neural network (CNN), which receives a video frame (image) downscaled to 96x176 as input and produces a three-channel matrix (120) of size 96x176 as output. The first channel predicts (contains, stores) the board mask, the second channel predicts the board edge mask, and the third channel predicts the board corner points (probability spots of pixels around the corner of the board, where the highest intensity values ​​are at the center of the corner and decrease toward the edges of the probability spot).

[0055] In some implementations, the following steps are performed to separate closely spaced boards (after processing the video frame using a convolutional neural network and obtaining the three-channel matrix described earlier): the board edge mask is subtracted from the board mask, then the resulting image is binarized according to a given threshold, after which the contours of all large objects are searched for in this image, and then these contours are approximated (bounding boxes are detected / determined) by quadrangles (simple geometric shapes that approximately correspond to the contour of the object, limited by four corners).

[0056] Some implementations use machine vision methods, such as those from OpenCV, to find object contours.

[0057] Some implementations use the findContours algorithm from the OpenCV computer vision library to find contours. cv::findContours(mask, contours, cv::RETR_TREE, cv::CHAIN_APPROX_SIMPLE). The algorithm traverses a binary image and finds the boundaries of objects, converting them into a sequence of points that form contours. It uses a traversal algorithm: it moves through adjacent pixels until it returns to the starting point, thereby forming the object's contour.

[0058] In some implementations, a Hough transform and / or a Canny edge detector are used to find the edges of objects.

[0059] In some embodiments, to determine the coordinates of the corners of chessboards (the corner points of the boards) for the obtained (defined, identified) quadrangles approximating the boards (described earlier in the section on separating closely located boards), the non-maximum suppression / NMS method / technique is used, which allows, in cases where the same object is selected during approximation / selection of boundaries by several bounding boxes, to determine the most probable / best bounding box of the object.

[0060] In some embodiments, to refine the coordinates of the corner points of the boards, for each vertex of each found quadrangle (the frame that bounds the board), the closest point to it is found, obtained from the third channel of the matrix (120), obtained after processing by the neural network (110).

[0061] In some embodiments, to determine the active board, the board within which the image center point is located is determined (identified), and if there is no such board, then the board with the largest area is selected.

[0062] A fragment of a video frame corresponding to an active chessboard is determined (130). In some embodiments, the following actions are performed to determine a fragment of a video frame corresponding to an active chessboard (the block corresponding to these actions is illustratively shown in Fig. 1 - 130):

[0063] • Scale the coordinates of the corners of the active chessboard to the size of the resulting video frame;

[0064] • Cut out (determine, find) the image of the active chessboard on the received video frame and, using perspective transformation, convert it into a rectangular image with a predetermined size (for example, 256x256 pixels).

[0065] In some implementations, the image of the active chessboard is cropped with an offset that can be predetermined, dynamically determined (e.g., defined by a formula), and / or randomly (within a predetermined range). In some implementations, the image of the active chessboard is taken (located, determined) with an offset that can be predetermined, dynamically determined (e.g., defined by a formula), and / or randomly (within a predetermined range). This approach is used to mitigate errors in finding the chessboard outline, camera shift, and also to capture fragments of pieces that extend beyond the edges of the board.

[0066] A multi-channel matrix (160) of the representation of the playing field and the figures located on it is formed by processing the image of the active chessboard (140) obtained in the previous step by a pool of machine learning models (150)

[0067] In some implementations, each time the multichannel matrix is ​​changed, it is stored in memory (e.g., for further analysis).

[0068] Some implementations use a pool of multiple models with different architectures and trained with different parameters.

[0069] In some embodiments, the multichannel matrix (160) is a nine-channel 8x8 matrix corresponding to a chessboard. Four channels of the matrix represent four board classes: empty board, white piece, black piece, and hand blocking. The remaining five channels represent piece types: pawn, knight, bishop, rook, and king / queen.

[0070] In some embodiments, the architecture of each model in the pool (150) consists of two parts:

[0071] • Backbone network;

[0072] • The head part of the neural network.

[0073] In some embodiments, the extraction of basic features from an image is performed using a basic neural network, for example, neural networks of the EfficientNetBO, EfficientNetBl, EfficientNetB2, EfficientNetB3, EfficientNetB4, EfficientNetB5, EfficientNetB6, EfficientNetB7, MobileNetv2, MobileNetV3Large, MobileNetV3 Small type and others can be used.

[0074] In some implementations, the base network may be any neural network, including a convolutional neural network, whose input is an RGB image and whose output is a feature map—a matrix containing key information about the input image. In some implementations, the base network is trained for the task of recognizing chess pieces on a chessboard. In some implementations, the training process is structured as follows: a dataset (training dataset) is pre-assembled from real chess game data, maximizing the diversity of data with different lighting conditions, piece sets (Staunton and others), board types, and other factors. Synthetic data is additionally generated using a 3D object development environment (e.g., Blender). 3D models of the chess pieces and boards are then prepared. In some implementations, various lighting conditions are simulated.In some implementations, the probabilities of pieces occupying specific positions on the chessboard are set to obtain diverse arrangements. In some implementations, the possible camera angles from which the board can be "viewed" are configured. In some implementations, other adjustable parameters are fixed and the generation of synthetic data is initiated. After the generation step, the output is a large number of different images of chessboards with pieces, overlaid with a background randomly selected from a large image dataset, such as the COCO dataset (https: / / cocodataset.Org / #home). In some implementations, real and synthetic data are combined into a single dataset, various neural network architectures are created, training parameters are set, quality metrics are defined, augmentations are added, and training is initiated.

[0075] The main neural network receives the resulting feature map (from the base network) as input, randomly drops out a certain percentage of neurons in the network, splits the feature map into two neural network branches, each of which sequentially identifies visual features at different levels of abstraction, adds nonlinearity, and finally merges the feature maps along the final axis, which determines the number of channels. This results in nine output channels, representing the probabilities of cell types and chessboard piece types.

[0076] The last axis (for example, in the concatenate operation in the context of neural networks) refers to using the last dimension of the tensors for the combination. In the context of multidimensional arrays (tensors) in mathematics and programming, an "axis" (or "dimension") refers to a specific direction in data space. Tensors can have multiple axes (dimensions). For example, a 2D array (matrix) has two axes: the first axis for the rows and the second axis for the columns. A 3D array adds a third axis, which can be interpreted as depth or height, depending on the context. Concatenate along the last axis combines the tensors in a way that expands the tensor along the last dimension. For example, if you have two 3D tensors with shapes (a, b, c) and (a, b, d), and concatenate along the last axis, the result will be a tensor of shape (a, b, c + d).Using the last axis allows the data structure in other dimensions to remain unchanged, changing only the size in a specific direction. This is important in many deep learning applications, where the shape of tensors has specific implications for interpreting data such as images, time series, or text.

[0077] In some embodiments, the main neural network is structurally composed of: three layers, described below: 1. The first layer is Dropout.

[0078] 2. The second layer, including two branches of neural networks.

[0079] The first branch of the neural network is responsible for field types, while the second branch is responsible for shape classes. Each branch contains a Conv2D convolutional layer, or a two-dimensional convolutional operation. This operation involves applying filters (or kernels) to the input image to extract features such as edges, textures, and corners. The filters traverse the entire image, calculating the dot product between the image pixel values ​​and the filter values, allowing the network to learn to identify various visual features at different levels of abstraction. Following the Conv2D in each branch is an Activation function—a nonlinear transformation applied element-by-element to the input data. This is an important component in neural network architecture, responsible for introducing nonlinearity into the learning process. These functions determine how the neuron should respond to the sum of the input signals.In our case, we use the softmax activation function (https: / / www.deeplearningbook.org / contents / mlp.html), which allows us to transform the input signal from the previous Conv2D layer into output matrices with a probability distribution, here - the types and colors of chess pieces.

[0080] 3. The final layer of the main neural network is the concatenate layer—a fusion operation that combines data from different layers or sources. This function takes input data arrays (e.g., the outputs of different layers of the neural network) and combines them into one, connecting them along a specified axis—in our case, the final axis, which corresponds to the number of channels in each branch of the neural network. The checkerboard image, passing through the serial connection between the base neural network and the main neural network, is transformed into the output layer.

[0081] The output layer is obtained by sequential processing by the base and head neural networks, where the base neural network extracts key information from the image, and the head neural network randomly excludes neurons, identifies visual features at different levels of abstraction, and adds nonlinearity. Each element of the channels (multichannel matrix 160) of the neural network's output layer represents the probability distribution of the occurrence of chess pieces: black, white, a hand, an empty square, a pawn, a knight, a bishop, a rook, a queen, or a king, on an 8x8 matrix corresponding to the chessboard, corresponding to each channel sequentially or in any order. The channels of the neural network's output layer, sometimes called filters, in this case represent the data obtained in the final layer of the neural network. Each channel is an 8x8 matrix whose cells directly correspond to the squares of the chessboard. Each channel, that is, each matrix, contains real values ​​ranging from 0 to 1.Each value represents the probability of a chess piece of a certain type and color being present on a square on the chessboard.

[0082] In some implementations, key factors in selecting a base network for use in the pool include depth, number of parameters, throughput (or, in other words, network runtime speed), and the ability to efficiently extract useful features at various levels of abstraction from input data. In some implementations, the base network may be a convolutional neural network, which is lightweight enough and optimal for use in mobile devices and embedded systems, robust in feature extraction to changes in lighting, scale, perspective, and other input data distortions. The architecture of such a neural network should be able to easily integrate with various main neural networks for specific recognition tasks.

[0083] Also, to speed up the training process, the base network is usually pre-trained on a large amount of data, and the process of adapting it to specific tasks requires minimal additional effort.

[0084] In some implementations, blocks that can be used in the underlying networks (and combinations thereof) are: Residuals, Inverted Residuals and Linear Bottlenecks, Depthwise Separable Convolutions, Squeeze-and-Excitation, Skip Connections.

[0085] During quality control on test data, the recognition quality is looked at, which is usually expressed as the percentage of correct identifications of the type and position of figures.

[0086] In some implementation variants, based on the recognition quality indicator, different configurations of neural networks are compared and those models (neural networks) that have different properties / structure but show better quality are selected into the pool (ensemble).

[0087] Examples of properties of models (neural networks) that may be taken into account when selecting into a pool (ensemble), depending on the neural network used, but not limited to:

[0088] • The alpha parameter in basic neural networks such as MobileNet is the network width coefficient. This parameter allows dynamic adjustment of the network width, i.e. the number of channels (or filters) in convolutional layers throughout the network architecture;

[0089] • B - serial number in the array EfficientNetBO, EfficientNetBl, EfficientNetB2, EfficientNetB3, EfficientNetB4, EfficientNetB5, EfficientNetB6, EfficientNetB7;

[0090] • Regularization power - a coefficient that is used to reduce overfitting by adding penalties to the weights of the convolutional layers in the model. Regularization can help improve the generalization ability of the model by preventing it from overfitting to the training data;

[0091] • dropout rate (de rate) – the percentage of random exclusion of neurons in the network;

[0092] • regularization coefficients (k_11);

[0093] • L2 regularization coefficient (k_12).

[0094] The backbone architectures of MobileNet and EfficientNet are quite different from each other, so if MobileNet is selected for one neural network, then to compensate (increase diversity in different types of neural networks, and therefore improve the overall quality), EfficientNet will be added to the pool.

[0095] In some implementations, EfficientNet is selected from the following pool: EfficientNetV2B0, EfficientNetV2Bl, EfficientNetV2B2, EfficientNetV2B3

[0096] 'de rate' controls the "dropout rate" parameter in the Dropout layer of the neural network. k_ll=0' refers to the L1 regularization coefficient, which is applied to the weights of the Conv2D and DepthwiseConv2D layers of the model. L1 regularization (also known as Lasso regularization) helps prevent model overfitting by adding a penalty for large weights equal to the absolute value of the weights multiplied by the coefficient k_1 . This causes some weights to be set to zero and can contribute to a sparser model. The value k_ll=0' signifies the absence of L1 regularization, since the multiplication coefficient is zero, and therefore no L1 penalty is applied to the weights.

[0097] 'k_12' in this context stands for L2 regularization, also known as Ridge regularization. The 'k_12' value determines the L2 regularization coefficient applied to the weights of the Conv2D and DepthwiseConv2D layers in the neural network. L2 regularization seeks to reduce the magnitude of the model weights (coefficients) by adding a penalty proportional to the sum of the squares of the weights to the loss function. This process helps combat overfitting by limiting the model's complexity, making the weights smaller in absolute value, leading to smoother and more stable predictions. If 'k_12' is 0, it means that L2 regularization is not applied to the weights of the corresponding layers. Increasing the 'k_12' value enhances the effect of regularization, forcing the model to further reduce the weight values ​​to minimize the regularization penalty in addition to minimizing the main loss function.

[0098] For example, by creating two different models (neural networks) with the following parameters:

[0099] B: 1 do_rate: 0.5 k_ll: 0 k_12: 0.00001

[0100] B: 3 de rate: 0.0 k_ll: 0 k_12: 0

[0101] Each neural network will show high quality results individually, and also high quality in an ensemble of neural networks.

[0102] That is, to select neural networks for an ensemble, they are selected in such a way that they are defined by different parameters, each have high quality, and the ensemble also has high quality.

[0103] Some implementations use a pool of neural networks instead of an ensemble. A pool refers to two or more separate neural networks, one of which is randomly or otherwise (e.g., in a round robin fashion) selected to process the data.

[0104] Quality is measured by comparing two matrices: a nine-channel 8x8 predicted matrix and a similar nine-channel gameboard representation matrix. In each channel, the cells directly correspond to chessboard squares and contain integer values ​​of 0 or 1. Each value represents the presence of a chess piece of a certain type and color on a chessboard square. 0 indicates the absence of a given class in a cell, 1 indicates the presence of a given class in a cell. The second matrix is ​​the neural network's prediction on the frame—a nine-channel output layer of the neural network, each channel element of which represents the probabilities of finding chess pieces of black, white, a hand, an empty square, a pawn, a knight, a bishop, a rook, a queen, or a king. Each channel is an 8x8 matrix, the arrangement of whose cells directly corresponds to the arrangement of chessboard squares. Each cell contains real values ​​ranging from 0 to 1.Each value represents the probability of a chess piece of a certain type and color being present on a square on the chessboard. def cells_acc(y_true, y_pred): return K.cast(K.equal(K.argmax(y_true[:, :, :, :4], axis=-l), K.argmax(y_pred[:, :, :, :4], axis=- 1)), K.floatx()) def figures_acc(y_true, y_pred): msk = K.cast(K.sum(y_true[:, :, :, :2], axis=-l), dtype='int64') return K.cast(K.equal(msk * K.argmax(y_true[:, :, :, 4:], axis=-l), msk * K.argmax(y_pred[:, :, :, 4:], axis=-l)),.

[0105] K.floatx())

[0106] Where cells_acc represents the accuracy of cell type predictions, and figures_acc represents the accuracy of figure type predictions, on each 8x8 cell.

[0107] Overall quality is measured as cells acc + figures acc

[0108] During inference, a neural network is randomly selected for each frame from a pool of pre-prepared neural networks.

[0109] In one embodiment of the method, each element of the first channel of the output layer of the set of neural networks represents the probability distribution of an empty cell on an 8x8 matrix corresponding to a chessboard.

[0110] In one embodiment of the method, each element of the second channel of the output layer of the set of neural networks represents the probability distribution of the locations of white chess pieces on an 8x8 matrix corresponding to a chessboard.

[0111] In one embodiment of the method, each element of the second channel of the output layer of the neural network set represents the probability distribution of black chess pieces on an 8x8 matrix corresponding to the chessboard. In one embodiment of the method, each element of the fourth channel of the output layer of the neural network set represents the probability distribution of a hand on an 8x8 matrix corresponding to the chessboard;

[0112] In one embodiment of the method, each element of the fifth channel of the output layer of the set of neural networks represents the probability distribution of a cell containing a pawn on an 8x8 matrix corresponding to a chessboard;

[0113] In one embodiment of the method, each element of the sixth channel of the output layer of the set of neural networks represents the probability distribution of a cell containing a knight on an 8x8 matrix corresponding to a chessboard;

[0114] In one embodiment of the method, each element of the seventh channel of the output layer of the set of neural networks represents the probability distribution of a cell containing a bishop on an 8x8 matrix corresponding to a chessboard;

[0115] In one embodiment of the method, each element of the eighth channel of the output layer of the set of neural networks represents the probability distribution of a cell containing a rook on an 8x8 matrix corresponding to a chessboard;

[0116] In one embodiment of the method, each element of the ninth channel of the output layer of the set of neural networks represents the probability distribution of a cell containing a queen or a king on an 8x8 matrix corresponding to a chessboard.

[0117] They analyze the arrangement of chess pieces on the playing field taking into account the rules of chess

[0118] In some embodiments, the analysis of the arrangement of chess pieces is carried out using a chess game engine (module), to which data about the current chess position is transmitted.

[0119] In some embodiments, the current chessboard layout and previous (current and one or more previous player moves / actions) previously stored in memory are analyzed. In some embodiments, the layout of all chessboard pieces or only a portion of them is analyzed. In some embodiments, each player move / action is stored. In some embodiments, each move is stored as an element of a linked list. In some embodiments, the entire chessboard layout is stored for each player move.

[0120] In some embodiments, the analysis of the chess piece arrangement includes, but is not limited to, the following chess situations: checkmate, stalemate, draw by the threefold repetition rule, draw due to insufficient pieces, draw due to exhaustion of the move limit, incorrect / illegal moves (e.g., a rook that made a diagonal move), "en passant capture", castling, "touch-move", "release hand - move made", etc.

[0121] Below are some comments regarding the situations analyzed.

[0122] A draw due to running out of moves includes the 50-move rule without a piece being captured or a pawn move, and the 75-move rule - if no pawn moves have been made or pieces captured in the last 75 moves, the game is declared a draw.

[0123] In some implementations, analysis of rules that require counting the number of moves may involve analysis of the current and previous moves of players, or analysis of the current move and a set of counters that is incremented each time such a move is detected.

[0124] The en passant rule: This is a rule that addresses specific situations where a pawn moves two squares from its starting position and another pawn can capture it en passant.

[0125] The "Touch, Move" rule. When a player touches a chess piece with the intention of making a move, they must move that piece if such a move is legal under the rules of the game. If a player intends to adjust the position of one or more pieces on the board, they must announce this in advance, usually by saying "adjusting" or a similar phrase before touching the piece. This prevents misunderstandings about their intentions.

[0126] The "Release your hand - move made" rule in chess determines the moment a move is completed. According to this rule, a move is considered completed when a player releases a piece after moving it to a new position on the board. If the player then removes their hand from the piece, they cannot change the move or move another piece. This rule helps establish clear and unambiguous conditions for completing a move and maintain order and fairness in the game.

[0127] In some implementations, recognized moves are recorded during the game, along with a text description of the events occurring during the game. In some implementations, the text description is presented in Portable Game Notation (PGN) or FEN format.

[0128] (https: / / en.wikipedia.org / wiki / Forsyth%E2%80%93Edwards_Notation).

[0129] In some implementations, after each completed move, the pgn string is processed by the mobile device, triggering heuristic analysis.

[0130] Heuristics are predetermined logic (analytical logic) for determining chess situations, such as 5-fold repetitions of moves. An example of a heuristic taken from the official rules of chess: The game is drawn if one or both of the following situations occur: the same position has appeared at least five times; positions are considered the same if and only if it is the same player's turn to move, pieces of the same name and color occupy the same squares, and the possible moves of all pieces of both players are the same.

[0131] Therefore the positions are not the same if

[0132] 1.1 at the beginning of the sequence of moves under consideration the pawn could have been captured en passant

[0133] 1.2 The king had the right to castle with a rook that was not moving, but lost it after its move. Castling rights are lost only after the king or rook has made a move.

[0134] Any series of at least 75 moves by each player was completed without advancing any pawns or capturing any pieces. If the last move resulted in checkmate, checkmate takes precedence.

[0135] The chess engine (module) begins its analysis of the game board with a given initial position of the pieces on the chessboard (virtual board, virtual game board; here and below, these terms are used interchangeably). Typically, this is the standard position of the pieces at the beginning of the game. In some implementations, upon receiving a video stream, the "virtual board"

[0136] Using a chess engine like Stockfish, a "virtual" game board is created that corresponds to this arrangement of pieces. A "virtual game board," in the context of engine-based chess analysis, refers to a digital representation of the chessboard and the arrangement of pieces on it. It is a software emulation of a real chess board, allowing the engine (module) to analyze the game, calculate moves, and evaluate positions. The virtual game board simulates the standard initial arrangement of pieces at the beginning of the game or any other specified arrangement of pieces. Using this representation, it is possible to conduct in-depth analysis of the game, determining optimal moves and identifying mistakes made by players. For each upcoming move, the engine provides a list of legitimate moves for the virtual game board. When processing new frames and receiving data on recognized squares of the real chess board, this data is compared with the list of legitimate moves.

[0137] If the analysis of the chess piece arrangement (chess game) reveals moves that do not correspond to any legitimate moves on the virtual game board, two copies of the current virtual board are created. On one board, the move is made virtually, changing the state of the virtual game board accordingly. On the second board, no changes are made virtually, ignoring the move (adding it to the list of ignored moves).

[0138] As new data (recognized moves) arrive, each virtual board can, in turn, spawn several child virtual boards with alternative outcomes. This creates a pool of virtual boards describing different possible outcomes for the real game.

[0139] Each virtual board in the pool is evaluated using a scoring function to determine its compliance with the state recognized by the neural networks over the entire game. The scores are adjusted during the processing of each frame.

[0140] The evaluation function is formulated as follows: To compare two virtual boards, two 8*8*n matrices are used, corresponding to each board, where n are classes, for example, pawn, rook, etc. The first matrix is ​​a representation of the virtual chessboard generated by the chess engine. The cells of such a matrix contain the values ​​either 0 or 1. For the parameter n = 9, when there are classes: empty square, white piece, black piece, blocking by hand, pawn, knight, bishop, rook, king / queen - the dimension of such a matrix will be equal to 8*8*9, i.e. the matrix will contain 9 channels of size 8*8. In each channel, the cells will directly correspond to the chessboard cells and contain the integer values ​​0 or 1. Each value will carry the meaning of the presence of a chess piece of a certain type and color on a square of the chessboard. 0 is the absence of a given class in a cell, 1 is the presence of a given class in a cell.The second matrix is ​​the neural network's prediction for the frame—a nine-channel output layer of the neural network, each element of which represents the probabilities of finding a black chess piece, a white chess piece, a hand, an empty square, a pawn, a knight, a bishop, a rook, a queen, or a king. Each channel is an 8x8 matrix, the arrangement of whose cells directly corresponds to the arrangement of squares on the chessboard. Each cell contains real values ​​ranging from 0 to 1. Each value represents the probability of a chess piece of a certain type and color on a square on the chessboard. To compare two virtual boards, the scoring function evaluates the similarity of two corresponding three-dimensional 8*8*n matrices to each other in the current frame. The first step is to multiply them element by element, obtaining a matrix K of the same size (8*8*n), and the second step is to sum all the elements of the matrix K into an integer A.High values ​​of the number A will correspond to strong similarity between the two matrices, while low values ​​of the number A will correspond to weak similarity between the two matrices. This process is repeated for each subsequent frame, calculating an exponential smoothing of the A values ​​as a result of a special evaluation function.

[0141] The device's RAM stores the highest-rated scenarios, which can be used later if differences arise between the state of the main virtual board and the virtual board obtained through neural network recognition. Since the highest A value for a given virtual board and the current real board reflects the greatest similarity between the two chess game states, the virtual board with the highest current score is used as the main virtual board, whose moves are displayed in the program interface during the game.

[0142] When conflicts between the main virtual board and the real board state recognized by neural networks occur, repeating over a predetermined number of frames, this is identified as an error in the move analysis algorithm, and an attempt is made to resolve the situation. The number of consecutive frames during which conflicts must occur to determine an error in the move analysis algorithm is predetermined based on the frequency of errors. Low values ​​for this parameter will result in frequent false alarms, while high values ​​will result in insufficient sensitivity in detecting errors in the move analysis algorithm. In our case, this parameter is set to 10 frames.

[0143] One way to resolve this situation is to roll back to a virtual board where no errors have been recorded in the past, try all possible moves with a search depth of 2-3 moves, and calculate a score using a scoring function for each of the options tried. If this search succeeds in finding a virtual board state that is consistent with the predicted state of the real board, then this virtual board is added to the pool of existing virtual boards.

[0144] If none of these methods resolve the situation, an invalid move is considered to have been made and an arbitrator's intervention is required. In some implementations, a corresponding notification is sent to the arbitrator's mobile device and / or the administrative panel (dedicated server).

[0145] In some embodiments, in response to the detection of one or a combination of predetermined chess situations, a corresponding indication or triggers / procedures corresponding to the situation are carried out.

[0146] In some implementations, the trigger is an indication and / or a set of actions. In some implementations, each video frame processed (according to the described method), the recognized placement of chess pieces on the playing field, and the results of the analysis of the chess piece placement, taking into account the rules of chess, are transmitted (from the mobile device) to a remote server for broadcast.

[0147] In some implementations, each video frame processed (according to the described method), the recognized chess piece placement on the playing field, the results of the chess piece placement analysis taking into account the rules of chess, and the video stream are transmitted (from the mobile device) to a remote server for broadcasting. The video stream is transmitted asynchronously. Some implementations use timestamps to synchronize the video stream and other transmitted data.

[0148] In some implementations, the initial state of the virtual playing field is transmitted (from the mobile device) to the remote server, and then the data of the current move, for example, in pgn notation.

[0149] In some implementations, the state of the virtual playing field and the current move data are transmitted (from the mobile device) to the remote server, for example, in pgn notation.

[0150] In some implementations, the remote server receives data transmitted from the mobile device and carries out its further transmission (broadcast) to other devices.

[0151] In some implementations, the remote server is configured to request a video stream from a mobile device. In some implementations, the entire video stream or individual fragments of it are requested. In some implementations, this can be used, for example, to display a disputed point on video to the referee.

[0152] In some embodiments, chess clocks are recognized using known means, such as computer vision or pre-trained machine learning models.

[0153] In some embodiments, the method for implementing a video assistant to a chess arbiter and broadcasting chess games includes the following steps:

[0154] Receive a video stream;

[0155] They determine and recognize the active chessboard in the video stream and form a virtual playing field corresponding to its state;

[0156] They analyze the players' actions with the chess pieces on the chessboard in the video stream and track the chess pieces on the chessboard, updating the state of the virtual playing field with each player's move;

[0157] In some implementations, player actions include chess moves (moving a piece on the board), touching a piece (capturing a piece), moving a piece, releasing a piece, and pressing the chess clock (if present).

[0158] In some embodiments, the detection (identification, recognition) of a touch, a player grabbing a piece, moving a piece, releasing a piece, touching a piece, pressing a chess clock is carried out using computer vision (e.g., OpenCV)

[0159] In some implementations, piece tracking is performed on the board when analyzing the chess piece arrangement on the playing field. Tracking refers to tracking a piece's movement across the chessboard without re-recognizing it. One implementation is described below. Since the initial piece arrangement at the start of the game is known, all piece tracking can be performed based solely on the color changes of the pieces in each square of the board (if a black piece disappears from one square and a black piece appears in another, there is an unambiguous understanding (unambiguous interpretation, identification) of which piece made the move, without relying on the recognition of the piece's denomination). For most of the game, piece movements are recorded by a combination of recognized square types (the first four output channels of the neural network: black / white / empty / hand) and an analysis of possible moves (by the chess engine).However, there are several situations where the data responsible for the piece type (the last five output channels of the neural network: pawn / knight / bishop / rook / queen-king) is used:.

[0160] • Pawn promotion. When a pawn reaches the opponent's last square and a player replaces it with a piece of their choice, the neural network's prediction channels responsible for the piece type are examined. Based on the recognized piece values ​​and the position of all pieces on the previous move, the chess engine is fed the recognized position of the pieces after promotion as the initial position. This position serves as the starting point for all logic processing until the end of the game.

[0161] • Recognition from any position. In situations where recognition must begin from an arbitrary position, neural network prediction is used, taking into account all channels of the output layer. Once a position is recognized, it is fixed as the initial position and passed on for analysis (for example, to a chess engine). From there, the standard recognition and processing logic continues.

[0162] In response to a change in the virtual playing field, the arrangement of chess pieces on the virtual playing field is analyzed, taking into account the current and previous moves of the players and their compliance with the rules of chess;

[0163] If the first move in a game is analyzed, then for this situation the previous moves are not analyzed.

[0164] In response to the detection of pre-defined chess situations, the corresponding trigger is executed;

[0165] In some implementations, a trigger may include an indication (e.g., displaying a dialog box, displaying a caption, or another visual component, such as an icon), as well as certain preset actions, such as, but not limited to, stopping a video stream, stopping broadcasting / digitizing, sending messages or notifications to a remote server or other device. In some implementations, triggers are implemented using programming languages ​​(JavaScript, Lua) and / or stored procedures. The current state of the virtual game board is transmitted to a remote server for broadcasting and / or digitizing of player moves.

[0166] In some embodiments, all information about the position of each piece for each move is transmitted to a remote server. In some embodiments, information about the current players at the chessboard (e.g., their identifiers, names, and other information), the game number, or its identifier is transmitted from the device (400) to the remote server (410). In some embodiments, before the start of the game, a game session is formed on the remote server (410), the player data is specified, and a session (game, game) identifier is assigned. The device performing the steps of the method (400) will be associated with such a session, and the remote server (410) will associate the state of the game board transmitted from the device (400) with this session. In some embodiments, the broadcast from the remote server (410) to the devices of spectators (411-413) is carried out in accordance with the sessions selected by the spectators.For example, user 411 may choose to view two games between different players simultaneously, while users 412 and 413 may choose to view only one. In some embodiments, the broadcast includes displaying the current positions of the chess pieces and the player's current move. In some embodiments, the broadcast may additionally include a video stream (video of the current game) or selected frames from such a video stream. In some embodiments, an analysis of the chess game may also be broadcast in accordance with the scenarios described earlier. In some embodiments, device 400 broadcasts real-time video of the current game or selected frames (e.g., every 5th, 10th, or other frame). In some embodiments (Fig. 6), two or more games in a competition / tournament may be shown to the viewer simultaneously, with the remote server asynchronously transmitting information on each game (each player's move) to the viewer's device.

[0167] In some broadcast implementations, the viewer has the ability to see all the players' moves in chess notation (Fig. 5). In some implementations, the viewer can "rewind" (go back / forward) or "play" the game from a certain previous move—that is, displaying a specific move (previously made) and the state of the chessboard at that moment. In some implementations, the current player's move is visually highlighted in the list of players' moves (Fig. 5).

[0168] In some implementations, players' moves in a game are further analyzed to identify the most successful / spectacular moves / combinations. In some implementations, the identified successful / spectacular moves / combinations are obtained through a heuristic analysis of the players' moves (either the entire game or a fragment thereof). The corresponding video fragment is then extracted from the video stream corresponding to these moves / combinations (e.g., by requesting only this fragment for transmission from a mobile device to a remote server), and audio / video effects can be added. This video fragment can then be broadcast from the remote server to viewers' devices connected for broadcasting.

[0169] In some embodiments, a remote server receives connections from users interested in broadcasting a chess game. In response to the connection, the server transmits information about the current state of the chessboard and the current player's move and / or previous moves. In one exemplary embodiment, a user with a mobile device (400) equipped with a camera implements a method for video assistant chess arbiter and broadcasting chess games, in which the current state of the virtual playing field is transmitted via communication channels (wired / wireless) to a remote server (410) for broadcasting and / or digitizing the players' moves. The server (410) then broadcasts the information to viewers (411-413) on their devices (computers, laptops, smartphones, smartwatches, or other smart devices) via communication channels.

[0170] It should be understood that the above description is illustrative and not restrictive. Many other embodiments will become apparent to those skilled in the art upon reading and understanding the above description. Therefore, the scope of the invention is determined by reference to the appended claims, as well as the full scope of equivalents to which such claims give rise.

[0171] The foregoing description sets forth numerous details. However, it will be apparent to one skilled in the art that aspects of the present invention may be practiced without these specific details. In some instances, to avoid obscuring the present invention, well-known structures and devices are shown in block diagram form rather than in detail.

[0172] It should be noted that, unless otherwise specifically stated, as will become apparent from the subsequent discussion, throughout the description, terms in the discussion such as "receiving," "determining," "selecting," "storing," "analyzing," etc., refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities in the registers and memory of the computer system into other data similarly represented as physical quantities in the memory or registers of the computer system or other such information storage, transmission, or display devices.

[0173] The present invention also relates to a device for performing the operations described herein. This device may be specially designed for the required purposes or may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored on a computer-readable storage medium, such as disks of any type, including floppy disks, optical disks, compact disks and magneto-optical disks, read-only memories (ROM), random access memory (RAM), programmable read-only memories (EPROM), electronically erasable read-only memories (EEPROM), magnetic or optical cards, or any type of media suitable for storing electronic instructions, each of which is connected to a computer system bus.

[0174] The algorithms presented herein are not inherently tied to a specific computer or other device. Various general-purpose systems can be used with programs in accordance with the teachings provided herein, or it may be more convenient to create a more specialized device to perform the required method steps. The required structure for various such systems will be as described in the description. Furthermore, aspects of the present invention are not described with reference to a specific programming language. It should be noted that various programming languages, as described herein, can be used to implement the provisions of the present invention.

[0175] Embodiments of the present invention may be provided in the form of a software product or software comprising a machine-readable medium with instructions stored thereon that can be used to program a computer system (or other electronic devices) to perform a process in accordance with the present invention. Machine-readable medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, machine-readable (e.g., computer-readable) medium includes machine-readable (e.g., computer-readable) storage media (e.g., read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices).

[0176] The words "example" or "exemplified" as used herein mean an example, instance, or illustration. Any aspect or solution described herein as an "example" or "exemplified" is not necessarily to be construed as preferred or advantageous over other aspects or solutions. Rather, the use of the words "example" or "exemplified" is intended to present concepts from a practical perspective. When used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." Furthermore, use of the term "implementation," "one embodiment," "implementation example," or "one implementation example" throughout the text does not mean the same embodiment or implementation example unless specifically described as such. Furthermore, the terms "first," "second," "third," "fourth," etc.The symbols used in this document are intended to designate different elements and do not necessarily have an ordinal value in accordance with their numerical designation.

[0177] While many changes and modifications of the invention will no doubt become apparent to one of ordinary skill in the art, it should be understood after reading the foregoing description that any particular embodiment shown and described by way of illustration is in no way to be considered limiting.

[0178] Therefore, references to details of various embodiments are not intended to limit the scope of the claims, which in themselves contain only features considered as a disclosure of the invention.

Claims

Formula 1. A method for recognizing and analyzing chess moves using a computer device comprising at least a processor, random access memory, and machine-readable instructions, comprising the following steps: • Receive a video stream; • Divide the video stream into video frames; • Determine the coordinates of the corners of the chessboards on each video frame and determine the active chessboard; • For each video frame, determine the fragment corresponding to the active chessboard; • For each video frame, a multi-channel matrix of representation of the playing field and the pieces located on it is formed by processing the fragment of the video frame of the active chessboard obtained in the previous step using a machine learning model or a pool of such models; • In response to a change in the multi-channel matrix of the representation of the playing field relative to the previous video frame, the arrangement of the chess pieces on the playing field is analyzed taking into account the chess rules; • In response to the identification of pre-defined chess situations as a result of the analysis, the corresponding trigger is executed.

2. The method according to paragraph 1, in which the predetermined chess situation may be one of the following situations: checkmate, stalemate, draw, draw by the rule of threefold repetition, draw due to insufficient pieces, draw due to exhaustion of the limit of moves, incorrect moves, "en passant capture", castling, "touch-move", "release hand - move made".

3. The method according to i.1, in which, in order to determine the coordinates of the corners of chessboards in a video frame, the chessboards are determined, closely located chessboards are separated and the contours of the chessboards are obtained, each of which is approximated by at least one rectangle, after which the NMS method is applied and a weighted averaging of the coordinates of the closely located corner points is carried out.

4. The method according to paragraph 3, in which the chessboards are determined on the received video frame as follows: • Reduce the size of the received video frame, bringing it to a pre-set size; • Obtain a board mask, a board edge mask, and board corner points by processing the video frame reduced in the previous step using a convolutional neural network; 5. The method according to paragraph 3, in which, in order to separate closely located chessboards, the board edge masks are subtracted from the board mask, the resulting image is binarized according to a predetermined threshold, after which the resulting contours are approximated by quadrangles.

6. The method according to claim 1, wherein the multi-channel matrix of the playing field is a nine-channel 8x8 matrix, where 8x8 is the representation of the playing fields, four channels provide the representation of four classes of the field: empty field, white piece, black piece, covered by a hand, and the remaining five channels provide the representation of the type of piece: pawn, knight, bishop, rook, king / queen.

7. A method for implementing a video assistant to a chess arbiter and broadcasting chess games using a computer device including at least a processor, RAM, and machine-readable instructions, includes the following steps: • Receive a video stream; • Determine and recognize the active chessboard in the video stream and form a virtual playing field corresponding to its state; • Analyze the actions of players with pieces on the chessboard in the video stream and track the chess pieces on the chessboard, updating the state of the virtual playing field with each move of the players; • Analyze the arrangement of chess pieces on the virtual playing field, taking into account the current and previous moves of the players and their compliance with the rules of chess; • In response to the identification of pre-defined chess situations as a result of the analysis, the corresponding trigger is executed; • Transmit the current state of the virtual playing field to a remote server to broadcast and / or digitize the players’ moves.

8. The method according to item 7, wherein the trigger comprises a predetermined indication and a predetermined action or set of actions.

9. The method according to paragraph 7, in which the predetermined chess situation may be one of the following situations: checkmate, stalemate, draw, draw by the rule of threefold repetition, draw due to insufficient pieces, draw due to exhaustion of the limit of moves, illegal moves, "en passant capture", castling, "touch l-move", "release hand - move made".

10. The method according to item 7, which additionally involves identifying and recognizing a chess clock in a video stream, after which the actions of the players are linked to the chess clock.

11. The method according to item 7, in which the remote server is configured to determine the best moves in the game and to indicate and broadcast them.

12. The method according to i.11, in which the indication of the best moves is carried out by displaying a video fragment of the best moves with added video / audio effects.

Images