Intelligent billiard scoring device and method based on visual identification

Abstract

The invention discloses an intelligent billiard scoring device and method based on visual identification, and relates to the related technical field of image processing, and the device comprises the steps: collecting high-definition video stream data of a billiard table surface in real time; performing dual-channel parallel sampling processing on the high-definition video stream data to obtain a key frame detail image set and a motion capture image set; performing rapid feature recognition to obtain multi-dimensional billiard motion features; after a pre-trained intelligent billiard scoring model is initialized according to a field billiard rule, the initialized intelligent billiard scoring model is used for analyzing and judging the multi-dimensional billiard motion characteristics, the billiard motion state is judged, score calculation is carried out, and a real-time score judgment result is generated. The technical problem that in the prior art, the billiard motion trail cannot be accurately captured, target ball overlapping cannot be judged, and then the billiard scoring accuracy is insufficient is solved, intelligent billiard scoring is achieved, and the technical effect of improving the billiard scoring accuracy is achieved.

Description

Technical Field

[0001] The present application relates to the technical field related to image processing, and in particular to an intelligent billiards scoring device and method based on visual recognition. Background Art

[0002] Traditional billiards coordinate recognition is mainly divided into mathematical algorithm mode and deep learning algorithm mode. The first one requires algorithm adaptation for different environments, so it is difficult to carry out large-scale market application; the other one collects the camera image above the billiard table to identify the specific coordinates of the billiard, and then calculates the first collision ball according to the movement trajectory of the white ball, and then performs subsequent scoring logic. However, this method cannot fully meet the actual scoring needs, especially under the snooker rules, which require that the designated colored balls will not be defined as fouls if they are hit (for example, the colored balls should be hit at present, but all the colored balls are blocked and cannot be hit directly. At this time, the player is ready to hit the pink ball, but the white ball hits the black ball for the first time during the actual hitting process, which is a foul; another situation is that the player actually aims at the pink ball that can be hit directly, but the white ball hits the black ball next to it due to the swing of the stick during the hitting process, which is also a foul). Therefore, only judging the overlap of the white ball trajectory and the target ball cannot handle this situation, and it is impossible to realize real-time monitoring and automatic judgment of the complex and rapidly changing table and billiard states during the game, thereby affecting the accuracy of billiards scoring.

[0003] Therefore, in the current relevant technologies, there is a technical problem that it is impossible to accurately capture the trajectory of the billiard ball and determine the overlap of the target balls, which leads to insufficient accuracy in billiard scoring. Summary of the invention

[0004] The present application provides an intelligent billiards scoring device and method based on visual recognition, which solves the technical problem in the prior art that the billiards ball movement trajectory cannot be accurately captured and the target ball overlap cannot be judged, thereby resulting in insufficient accuracy in billiards scoring. Intelligent billiards scoring is realized, achieving the technical effect of improving the accuracy of billiards scoring.

[0005] The present application provides an intelligent billiard scoring device based on visual recognition, the device comprising: a high-definition video stream data acquisition module, used for collecting high-definition video stream data of a billiard table in real time through a high-performance image acquisition module; a parallel sampling processing module, used for performing dual-channel parallel sampling processing on the high-definition video stream data to obtain a key frame detail image set and a motion capture image set; a fast feature recognition module, used for performing fast feature recognition based on the key frame detail image set and the motion capture image set to obtain multi-dimensional billiard motion features; a real-time score determination result generation module, used for initializing a pre-trained intelligent billiard scoring model according to live billiard rules, and then using the initialized intelligent billiard scoring model to analyze and determine the multi-dimensional billiard motion features, determine the billiard motion state and perform score calculation, and generate a real-time score determination result.

[0006] In a possible implementation, the parallel sampling processing module also performs the following processing: based on the high-definition video stream data, extract the high-definition desktop image and perform semantic segmentation, remove the interfering background, and generate a desktop graphics collection; build a parallel sampling module, the parallel sampling module includes a detail capture channel and a motion capture channel; input the desktop graphics collection into the parallel sampling module, and the detail capture channel and the motion capture channel respectively perform key frame extraction and down-sampling to obtain the key frame detail image set and the motion capture image set.

[0007] In a possible implementation, the fast feature recognition module further performs the following processing: based on the key frame detail image set, extracting an original desktop image of a batting action cycle; based on the original desktop image, performing target object recognition, calibrating multiple targets and target starting positions; for the multiple targets, traversing the key frame detail image set and the motion capture image set in chronological order to extract multi-dimensional billiards motion features.

[0008] In a possible implementation, the fast feature recognition module further performs the following processing: for the multiple targets, traverse the motion capture image set in chronological order and extract the motion trajectory features of each target respectively; based on the motion trajectory features, extract trajectory mutation points, and match key detail images from the key frame detail image set according to the time mark of the trajectory mutation point; extract multidimensional detail features of the multiple targets according to the key detail images; and fuse the motion trajectory features and multidimensional detail features of each target to generate the multidimensional billiards motion features.

[0009] In a possible implementation, the real-time score determination result generation module further performs the following processing: collecting scoring rule data of various billiard game types, performing centralized analysis, and extracting common basic rule data as basic framework data; using the basic framework data to pre-build an intelligent billiard scoring model, the intelligent billiard scoring model includes a score determination unit and a foul detection unit; according to the on-site billiard rules, comparing and analyzing with the basic framework data, generating a rule adjustment vector, and adjusting the rule parameters of the intelligent billiard scoring model based on the rule adjustment vector.

[0010] In a possible implementation, the real-time score determination result generation module further performs the following processing: based on the initialized intelligent billiards scoring model, extract the multi-dimensional detail features of the original desktop image from the multi-dimensional billiards motion features; extract the target object area according to the multi-dimensional detail features of the original desktop image, including extracting the pocket coordinate area, the billiard ball coordinate area, the player's palm coordinate area, and the cue coordinate area; segment and calculate the target object area to generate the cue direction vector; based on the cue direction vector, combined with the player's ball distribution information, perform a hitting extension calculation to identify and obtain the target impact ball; based on the multi-dimensional billiards motion features, extract the white ball motion trajectory data, and identify and extract the first impact ball therefrom; based on the basic information of the first impact ball, perform a foul determination and score calculation in combination with the target impact ball and the player's ball distribution information to generate a real-time score determination result.

[0011] In a possible implementation, the real-time scoring determination result generation module further performs the following processing: based on the player's palm coordinate area, extracting the player's action detail features, performing action foul determination, and generating an action foul determination result; generating a batting order rule based on the target impact ball and the player's ball distribution information, matching the information with the first impact ball, and generating a rule foul determination result; obtaining the terminal position information of the first impact ball, performing pocket identification, and generating a pocket identification result; generating the real-time scoring determination result based on the action foul determination result, the rule foul determination result, and the pocket identification result.

[0012] The present application also provides an intelligent billiards scoring method based on visual recognition, the method comprising: using a high-performance image acquisition module to collect high-definition video stream data of a billiard table in real time; performing dual-channel parallel sampling processing on the high-definition video stream data to obtain a key frame detail image set and a motion capture image set; based on the key frame detail image set and the motion capture image set, performing rapid feature recognition to obtain multi-dimensional billiards motion features; after initializing a pre-trained intelligent billiards scoring model according to live billiards rules, using the initialized intelligent billiards scoring model to analyze and judge the multi-dimensional billiards motion features, determine the billiards motion state and perform score calculation, and generate a real-time score determination result.

[0013] The intelligent billiard scoring device and method based on visual recognition proposed in this application is intended to collect high-definition video stream data of the billiard table in real time; perform dual-channel parallel sampling processing on the high-definition video stream data to obtain a key frame detail image set and a motion capture image set; perform rapid feature recognition to obtain multi-dimensional billiard motion features; initialize the pre-trained intelligent billiard scoring model according to the on-site billiard rules, and use the initialized intelligent billiard scoring model to analyze and judge the multi-dimensional billiard motion features, determine the billiard motion state and perform score calculations, and generate real-time score determination results. It solves the technical problems existing in the prior art that the billiard motion trajectory cannot be accurately captured and the target ball overlap is judged, which leads to insufficient billiard scoring accuracy, realizes intelligent billiard scoring, and achieves the technical effect of improving billiard scoring accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solution of the embodiment of the present disclosure, the accompanying drawings of the embodiment of the present disclosure will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the device according to the embodiment of the present application. It should be understood that the previous or following operations are not necessarily performed precisely in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0015] Figure 1 A schematic diagram of the structure of an intelligent billiards scoring device based on visual recognition provided in an embodiment of the present application.

[0016] Figure 2 A schematic flow chart of an intelligent billiards scoring method based on visual recognition provided in an embodiment of the present application.

[0017] Explanation of reference numerals: high-definition video stream data acquisition module 10, parallel sampling processing module 20, fast feature recognition module 30, real-time score determination result generation module 40. DETAILED DESCRIPTION

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0019] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0020] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, device, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0021] The present application embodiment provides an intelligent billiard scoring device based on visual recognition, such as Figure 1 As shown, the device comprises: The high-definition video stream data acquisition module 10 is used to collect the high-definition video stream data of the billiard table in real time through a high-performance image acquisition module.

[0022] Preferably, a high-performance image acquisition module, that is, a high-quality image acquisition device (such as a high-definition camera) is used to continuously and dynamically monitor the billiard table during the billiard game, and high-definition video stream data of the billiard table surface is obtained in real time. Specifically, the image acquisition module is installed 2-3 meters above the billiard table or diagonally above the billiard table, and can shoot with high-definition (such as 1080p or higher resolution) or ultra-high-definition video quality to ensure that the billiard table surface and the movement trajectory of the billiard balls can be clearly captured, especially at high-speed movement, without blurring or frame loss, and high-bandwidth and high-quality signal transmission is required between the image acquisition module and the computer processing system to ensure stable transmission of real-time high-definition video streams and avoid data loss. For example, an RJ45 network port is used to communicate with the computer over the network, and a CCD image sensor chip (charge-coupled device, used to convert captured light signals into electrical signals to achieve image acquisition) is used to capture the billiard table screen, providing high-quality video stream data, such as a resolution and frame rate of 30 frames per second or higher, including image information of all billiard balls on the billiard table surface and their movements.

[0023] The parallel sampling processing module 20 is used to perform dual-channel parallel sampling processing on the high-definition video stream data to obtain a key frame detail image set and a motion capture image set.

[0024] Preferably, the high-definition video stream data is processed by two different image processing methods (i.e., dual-channel parallel sampling), and different types of image sets are extracted, including key frame detail image sets and motion capture image sets. Specifically, the video stream is processed simultaneously by two different processing paths (channels), that is, each channel focuses on different image feature extraction, and extracts different image information in the video stream respectively. For example, one channel is used to capture motion information in the image (such as the moving trajectory of the billiard ball), and the other channel is used for key frame details (such as the appearance, color, table surface details, etc. of the billiard ball). By analyzing the changes in the video stream in real time, frames at key moments are selected and analyzed in detail, such as extracting details such as the color, shape, and position of the billiard ball, and by analyzing the changes in consecutive frames in the video stream in real time, Extract motion information, such as performing motion detection, object tracking and trajectory analysis, capture the motion trajectory of the billiard ball, calculate the ball speed, determine the collision angle, etc., and then obtain a key frame detail image set and a motion capture image set, where a key frame refers to an image frame that represents a key moment or change point in a continuous video stream. For example, in a billiard game, the corresponding frame when the ball collides or stops, the key frame detail image contains high-definition image information of the billiard table and billiard ball, which helps to identify the specific position of the billiard ball, the state of the table, the ball logo, etc.; motion capture refers to extracting and tracking the motion trajectory of the billiard ball in the video stream, especially the dynamic changes of the billiard ball's speed, direction, collision, etc. Through the motion capture image set, it is possible to accurately identify the motion state of the billiard ball, determine the trajectory of the ball, predict the subsequent movement of the ball, and analyze whether key behaviors such as scoring have occurred.

[0025] The fast feature recognition module 30 is used to perform fast feature recognition based on the key frame detail image set and the motion capture image set to obtain multi-dimensional billiard motion features.

[0026] Preferably, according to the key frame detail image set and the motion capture image set, various feature information involved in the billiard motion process is identified and extracted through efficient image analysis and feature extraction algorithms, covering not only the static position of the ball but also the dynamic motion characteristics of the ball, and finally constructing multi-dimensional feature data reflecting the motion state of the billiard ball, including the number, position, color, table layout and motion trajectory of the billiard ball. Specifically, using efficient image recognition algorithms, key features that can describe the motion state of the billiard ball are extracted from the key frame detail image set and the motion capture image set, including edge detection recognition. Object detection and classification: detect and identify billiard balls, holes, obstacles on the table, track the trajectory of the ball, analyze the displacement, speed, collision and other motion characteristics of the billiard ball from hitting to stopping, obtain the color, size and other characteristics of the billiard ball, fuse the features in the key frame detail image and the motion capture image, combine the motion features of different dimensions through multi-dimensional data fusion technology, generate multi-dimensional billiard motion features, and realize a full range of analysis of billiard motion, including the static position of the billiard ball, trajectory prediction, collision judgment (speed and direction), scoring status judgment, etc.

[0027] The real-time score determination result generating module 40 is used to initialize the pre-trained intelligent billiards scoring model according to the live billiards rules, and then use the initialized intelligent billiards scoring model to analyze and judge the multi-dimensional billiards motion characteristics, determine the billiards motion state and perform score calculation to generate a real-time score determination result.

[0028] Preferably, the intelligent billiard scoring model is a deep learning model trained by a large amount of annotated data, which can automatically identify and analyze various features in billiard sports, such as ball position, motion trajectory, collision, pocket situation, foul, etc. The core of pre-training is to enable the model to accurately determine billiard scores, fouls, motion trajectory and other information in actual games, reduce manual intervention and achieve automation, wherein the annotated data content may include the coordinates of the billiard ball in the image (for example, the specific positions of the red ball, white ball and colored ball), the motion trajectory of the billiard ball (for example, The collision and rebound between the white ball and the target ball, etc.), pocketing situation (whether the ball is successfully pocketed, the order of pocketing and the score), foul events (such as failure to hit the target ball, foul shots, etc.), batting order (snooker requires players to hit the red balls and colored balls in order; 9-ball requires players to hit balls 1 to 8 in order, and finally pocket the 9th ball legally to win; Chinese 8-ball requires players to hit their target balls in order, such as balls 1-7 or balls 9-15, and finally pocket the 8th ball legally to win).

[0029] Preferably, the pre-trained intelligent billiards scoring model is initialized according to the live billiards rules, which means that the intelligent billiards scoring model is used to perform accurate score calculation and judgment according to snooker rules and real-time billiards motion data, and the game results are generated in real time. For example, initialization is performed according to the snooker rules, that is, the scoring model is set according to the rules to ensure that all scenarios in the snooker game can be recognized and processed. The snooker rules involve billiards hitting order, target balls, foul judgments, score calculations and other aspects. Specifically, in a snooker game, players must hit red balls and colored balls in the prescribed order. This set of rules is loaded during initialization, including how to distinguish between red balls and colored balls, how to hit colored balls after hitting red balls, etc.; in snooker, each ball has a different score, 1 point for red balls, and 2 points for colored balls according to their type (such as 2 points for yellow balls). During initialization, the score of each ball is set. In snooker, if a player violates the rules (for example, he does not hit the correct target ball, hits the ball but does not touch the ball, etc.), he will be judged as a foul. These rules are loaded during the initialization process, and the foul can be automatically identified during the game. During initialization, it is also necessary to set how to judge the status of the table, which balls have been pocketed, which balls are still on the table, and the changes after the player hits the ball. In addition, each time a player hits a red ball, he must first hit a red ball, and only after the hit is successful can he hit the colored ball. After the colored ball is hit successfully, the colored ball will be put back on the table. Until all the red balls are hit, the player needs to hit the colored balls in the prescribed order: first the yellow ball (2 points), then the green ball (3 points), hit them in turn, and finally the black ball (7 points).

[0030] Preferably, the initialized intelligent billiard scoring model is used to analyze the multi-dimensional billiard motion characteristics, evaluate whether each hitting action of the player complies with the rules and the specific scoring situation, such as judging whether the player hits the correct target ball according to the rules, whether the hit ball enters the pocket, etc. Specifically, it includes a scoring determination unit and a foul detection unit. The scoring determination unit is responsible for calculating the score based on the motion state of the ball and the rules. The snooker game scoring determination includes 1 point for a red ball entering the pocket, different points for different colored balls entering the pocket (yellow ball 2 points, green ball 3 points, brown ball 4 points, blue ball 5 points, pink ball 6 points, black ball 7 points), continuous hitting (such as hitting the red ball and the colored ball alternately The foul detection module is responsible for identifying and determining fouls in the game. Common fouls include players not hitting the correct target ball (such as directly hitting the colored ball when hitting the red ball), the white ball entering the pocket when hitting, not touching any ball after hitting the ball (or not hitting the target ball according to the rules), inappropriately hitting the ball in the same shot, and wrong hitting order (such as trying to hit the colored ball before all the red balls are hit). When a foul occurs, the foul detection module will provide real-time feedback and deduct points or impose penalties according to the snooker rules.

[0031] Preferably, based on real-time image analysis and physical models (such as ball rebound, collision, rotation, etc.), the state of the ball is evaluated according to the real-time collected motion data and rules, including determining whether the ball will enter the pocket by calculating the motion trajectory of the billiard ball, and updating the score in real time; determining the motion state of the white ball, if the white ball enters the pocket, immediately determining it as a foul; determining the relative positions of all balls on the table, and whether the order of pocketing complies with the rules; thereby predicting the final position, motion trajectory, etc. of the ball, and then making a score or foul judgment, and finally automatically calculating and outputting the score judgment result of the current game, that is, the score is updated every time a ball enters the pocket; if a foul occurs, the score is deducted according to the rules; and the score result will be displayed in real time on the game interface, players and spectators can directly see the current game score, and the current status of the game will also be updated, including the number of remaining balls, situation changes, etc., thereby realizing intelligent billiard shot calculation and providing more accurate scoring and game results.

[0032] Furthermore, the specific configuration of the parallel sampling processing module 20 also includes, based on the high-definition video stream data, extracting high-definition desktop images and performing semantic segmentation, removing interfering background, and generating a desktop graphics collection; building a parallel sampling module, the parallel sampling module includes a detail capture channel and a motion capture channel; inputting the desktop graphics collection into the parallel sampling module, and the detail capture channel and the motion capture channel respectively perform key frame extraction and down-sampling to obtain the key frame detail image set and the motion capture image set.

[0033] Preferably, a high-definition image of a billiard table is extracted from a high-definition video stream to ensure clear image quality and rich details, and semantic segmentation is performed on the high-definition table image, that is, each pixel in the image is classified to identify different areas. For example, semantic segmentation can distinguish which parts are the billiard table and which are the background or other interference areas, including identifying different areas such as the billiard table, billiard balls, ball bags, and cue sticks. Specifically, the area related to the billiards is separated from the background, and irrelevant information (such as audiences, lights, venues, and other interference backgrounds) is removed, focusing on the image part of the billiard table and billiard balls, thereby generating a desktop graphic collection, which includes the distribution information of the billiard table, billiard balls, and other important elements.

[0034] Preferably, a parallel sampling module is constructed, wherein the parallel sampling module refers to an image processing unit, which is responsible for parallel processing of input graphic data, and performs processing of multiple tasks simultaneously through parallel sampling to improve processing efficiency and accuracy. The parallel sampling module includes a detail capture channel and a dynamic capture channel. Specifically, the detail capture channel is used to capture static details in the image, such as the precise position of the billiard ball, the state of the table, the color and shape of the ball, etc., that is, the detail capture channel extracts the details in the image with high precision and generates high-resolution image information to identify the position of the ball, the table layout, the relative position of the ball, etc.; the dynamic capture channel is used to capture dynamic changes in the image, especially the movement process of the billiard ball, that is, extracting the dynamic information of each frame from the video stream, analyzing the movement, collision, rotation, etc. of the ball, and then generating a set of motion capture images.

[0035] Preferably, the desktop graphics collection is input into the parallel sampling module, and the detail capture channel is used to extract key frames, that is, to select those frames containing key information from the continuous video frames. Usually, the key frames represent important events or changes in the video. For example, in a billiards game, when the billiards collide, enter the pocket, and other key actions, the corresponding frames will be selected as key frames. The detail capture channel processes the desktop graphics collection to extract these image frames containing key information and generate a key frame detail image set; then the dynamic capture channel is used for downsampling, that is, the image is processed with a reduced resolution, thereby retaining the motion trajectory and dynamic information of the billiards, but the details may be relatively few, which is used to reduce the amount of calculation and improve the processing speed. The dynamic capture channel obtains a motion capture image set by downsampling the continuous frames in the video stream, that is, image data containing dynamic information such as the billiards motion process and the collision between balls. These image sets are used for static analysis (such as desktop layout, ball position) and dynamic analysis (such as billiards motion trajectory, collision, etc.), respectively, to provide key visual information for scoring judgment and motion analysis.

[0036] Furthermore, the specific configuration of the rapid feature recognition module 30 also includes: extracting an original desktop image of a batting action cycle based on the key frame detail image set; performing target object recognition based on the original desktop image, calibrating multiple targets and the starting positions of the targets; and for the multiple targets, traversing the key frame detail image set and the motion capture image set in chronological order to extract multi-dimensional billiards motion features.

[0037] Preferably, the key frame detail image set includes the preparation stage before hitting the ball, the impact stage and the stage after the ball enters the pocket. The original desktop image before the hitting action (i.e., the billiard table layout image) is extracted from the key frame detail image set, which usually refers to the position of all balls on the billiard table and the state of the table when the player is preparing to hit the ball. This original desktop image is the starting point of the entire hitting action; based on the original desktop image, target object recognition is performed based on a deep learning algorithm (for example, YOLO, FasterR-CNN and other target detection networks), that is, all relevant objects are recognized in the original desktop image, including the table pocket, table boundary, billiard balls (billiard ball positions, such as white balls, red balls and colored balls), the player's posture and action (hitting action and the direction of movement of the cue) and the cue used by the player (such as analyzing its direction and the contact point of the cue), and then the positions of these objects are calibrated according to the recognition results, and their starting states (for example, the initial position of each billiard ball, the direction of the cue, the player's position, etc.) are recorded.

[0038] Preferably, multiple targets (billiard balls, cues, players, etc.) are tracked, and the dynamic changes in the process of hitting the ball are restored by calculating the position of each object at each time point, and the key frame detail image set (including static images, representing the state of the billiard ball at different time points before and after hitting the ball) and the motion capture image set (dynamic images of billiard ball motion information, motion trajectory, collision, rebound, etc. of the billiard ball) are traversed in chronological order (i.e., the timestamp of each frame of the image), and then the position of the ball, the table layout, the relative position between the billiard balls, and the movement direction, speed, trajectory after collision, etc. of the ball are extracted, that is, the multi-dimensional motion characteristics of the billiard ball are obtained, including the number, position, color, table layout and motion trajectory of the billiard ball, so that the motion state of the billiard ball can be tracked and analyzed in real time, and accurate game judgment and automatic scoring can be achieved.

[0039] Furthermore, the specific configuration of the fast feature recognition module 30 also includes, for the multiple targets, traversing the motion capture image set in chronological order, and extracting the motion trajectory features of each target object respectively; extracting the trajectory mutation points based on the motion trajectory features, and matching the key detail images from the key frame detail image set according to the time marks of the trajectory mutation points; extracting the multidimensional detail features of the multiple targets according to the key detail images; and fusing the motion trajectory features and multidimensional detail features of each target object to generate the multidimensional billiards motion features.

[0040] Preferably, according to multiple targets (billiard balls, tables, cues, players, etc.) in a billiard game, by performing a time-sequential analysis on the motion capture image set, the motion trajectory features of each target are extracted respectively, including key data of the target's motion path, such as the position features of each target (such as billiard balls, cues, etc.) in each frame, the path of the target from the starting position to the current position, and the identification of dynamic information such as motion direction, speed, acceleration, etc.; then, according to the motion trajectory features, trajectory mutation points are extracted, including collisions between white balls and other billiard balls, entry of billiard balls into pockets, and sudden changes in direction or speed of billiard balls, each trajectory mutation point has a corresponding timestamp, and the corresponding image frames are found in the key frame detail image set according to these timestamps, reflecting the specific state of the target at that moment, and then the images corresponding to the trajectory mutation points are matched from the key frame detail image set according to the time stamp, which contain important details, such as the specific position of the ball, the direction of the cue, the action of the player, etc.

[0041] Preferably, multi-dimensional detail features of multiple targets are extracted from the key detail images, which may include target type, such as white ball, red ball, colored ball, etc., each ball has different rules and scores; color, balls of different colors represent different types of balls (such as red ball, yellow ball, black ball, etc.); number, each colored ball has a fixed number (such as red ball is 1 point, black ball is 7 points, etc.); pattern, some types of billiard balls (such as red balls and colored balls) may have different patterns or logos; finally, the motion trajectory features of each target and the multi-dimensional detail features are fused, which can not only understand the dynamic changes of each target during the motion process, but also identify the static information such as the type and color of each target, and finally generate complete multi-dimensional billiard motion features, including dynamic features, such as ball position, motion direction, speed, collision, etc., and static features, such as ball type (red ball, colored ball, etc.), color, number, pattern, etc., so as to track the game status in real time, accurately determine scores and fouls, and improve the automation level and accuracy of the game.

[0042] Furthermore, the specific configuration of the real-time score determination result generation module 40 also includes collecting scoring rule data of various billiard game types, performing centralized analysis, and extracting common basic rule data as basic framework data; using the basic framework data to pre-build an intelligent billiard scoring model, the intelligent billiard scoring model includes a scoring determination unit and a foul detection unit; according to the on-site billiard rules, comparing and analyzing with the basic framework data, generating a rule adjustment vector, and adjusting the rule parameters of the intelligent billiard scoring model based on the rule adjustment vector.

[0043] Preferably, different billiard games (such as snooker, eight-ball, nine-ball, etc.) have different scoring rules and operating procedures. For example, in snooker, red balls and colored balls are hit in a specific order, and each ball has a different score, while in eight-ball or nine-ball, the rules and scoring methods are completely different. Specifically, detailed information on scoring rules, foul rules, hitting order, etc. for each game type is collected from multiple sources (such as rule books, official rules, game video data, etc. of various billiard games), and the rules of these different game types are centrally analyzed, including analyzing the scoring of balls in various billiard rules. The basic framework data includes the player's batting order, target ball selection (such as hitting the red ball, hitting the designated colored ball, etc.), player foul definition (such as hitting the wrong target ball, the white ball going into the pocket, etc.), pocketing rules and pocketing judgment, etc.

[0044] Preferably, an intelligent billiard scoring model is pre-built based on the basic framework data, which is used to automatically judge and calculate scores and fouls, including a scoring judgment unit and a foul detection unit. Specifically, the scoring judgment unit is responsible for judging the score after the billiard ball enters the pocket. According to the rules of different games, the model calculates the score of each ball and judges whether the player hits it according to the rules, that is, the basic framework data is used as the basic rule input to judge the ball entering the pocket, and then according to the game rules, the player's batting order is analyzed, and the corresponding score is determined according to the type of ball hit, and continuous hitting is performed to score. Whenever a ball enters the pocket, the scoring judgment unit will immediately update the current score and provide feedback according to the rules. When a round ends (for example, the red ball hits the ball), the scoring judgment unit will automatically update the current score. After the shot is completed, the colored balls go into the pockets one by one), the scoring judgment unit will accumulate all the scores and prepare for the score calculation of the next round; the foul detection unit is used to detect whether the player has violated the rules of the game, such as whether the wrong target ball is hit, whether the white ball goes into the pocket, two shots, hitting the ball without touching any ball, etc., that is, the foul rules of the basic framework data are used as input, and common foul behavior types are predefined. Whenever a player hits the ball, the foul detection unit will determine in real time whether a foul has occurred, including whether the correct target ball is hit, the white ball goes into the pocket, two shots, no target ball is hit, etc. If a foul is detected, the foul detection unit will decide on the penalty measures according to the specific foul type, such as losing the right to hit the ball.

[0045] Preferably, the specific rules of the current game are compared with the constructed basic framework data to find the differences in the rules, including comparing the player's batting order, scoring method, foul determination, etc. in the rules, finding out which parts need to be adjusted according to the on-site rules, and generating a rule adjustment vector, which describes what adjustments need to be made to the intelligent billiards scoring model in the current game, including changes in scoring rules, foul determination rules, pocketing rules, etc. For example, a game may modify the scoring rules of specific balls (such as increasing or decreasing the scores of certain colored balls), or define foul behaviors differently (such as certain fouls do not deduct points). Finally, according to the rule adjustment vector, the rule parameters in the intelligent billiards scoring model are dynamically adjusted. For example, if the rule adjustment vector shows that the red ball score in a certain game needs to be increased, the scoring rules in the model will be updated. If the adjustment vector indicates that a certain foul behavior requires penalty points or increased penalties, the model will be modified according to the on-site rules, ensuring that the intelligent billiards scoring model can be automatically adjusted according to specific rules in different games, and accurate scoring determination and foul detection can be performed.

[0046] Furthermore, the specific configuration of the real-time score determination result generation module 40 also includes: based on the initialized intelligent billiards scoring model, extracting the multi-dimensional detail features of the original desktop image from the multi-dimensional billiards motion features; extracting the target object area according to the multi-dimensional detail features of the original desktop image, including extracting the pocket coordinate area, the billiard ball coordinate area, the player's palm coordinate area, and the cue coordinate area; segmenting and calculating the target object area to generate a cue direction vector; based on the cue direction vector, combined with the player's ball distribution information, performing a hitting extension calculation to identify and obtain the target impact ball; extracting the white ball motion trajectory data according to the multi-dimensional billiards motion features, and identifying and extracting the first impact ball therefrom; based on the basic information of the first impact ball, combining the target impact ball and the player's ball distribution information, performing a foul determination and a score calculation to generate a real-time score determination result.

[0047] Preferably, the initialized intelligent billiard scoring model is used to identify and extract multi-dimensional detail features of the desktop image from the original desktop image through image processing technology (such as computer vision and deep learning), including the type and number of balls (such as red balls, colored balls, white balls, etc.), the color and pattern of the balls (such as the color and pattern of different colored balls) and the position of the balls (the precise position of each ball on the desktop), and then the image processing technology is used to extract multiple target areas related to the game from the desktop image, which may include the pocket coordinate area (identifying and extracting the positions of the six pockets on the billiard table), the billiard coordinate area (extracting The image segmentation is then performed to segment the target area (i.e., image segmentation). The geometric features of each target object are further extracted, especially the cue direction vector (including accurate judgment of the cue’s orientation and angle), which indicates the contact direction between the cue and the white ball, i.e., the direction of the path the player plans to hit the white ball. This helps to predict the ball’s trajectory and determine the target ball to be hit.

[0048] Preferably, based on the cue direction vector and the player's ball distribution information, a hit extension calculation is performed to understand the player's current hit target and calculate the direction and speed that the white ball will move after hitting. The hit extension calculation can predict the collision between the white ball and other billiard balls and identify the target impact ball. Then, based on the initial position of the white ball and the hitting direction of the cue, the motion trajectory of the white ball is extracted, including its speed, direction, acceleration and other dynamic data. By analyzing the motion trajectory of the white ball, it is determined which billiard ball the white ball hit during the first impact, that is, the first impact ball, which is helpful for foul detection (for example, whether the wrong target ball is hit) and score determination (for example, whether the correct ball is hit); finally, based on the basic information of the first impact ball, the identification of the target impact ball and the player's ball distribution information, it is determined whether the player hits according to the rules. If the player commits a foul, it is immediately determined and the corresponding score is deducted; if there is no foul, the score is calculated according to the billiard rules, and finally a real-time score determination result is generated, thereby greatly improving the automation level and accuracy of the game and reducing the possibility of human misjudgment.

[0049] Furthermore, the specific configuration of the real-time scoring determination result generation module 40 also includes: based on the player's palm coordinate area, extracting the player's action detail features, performing action foul determination, and generating an action foul determination result; generating a batting order rule according to the target impact ball and the player's ball distribution information, matching the information with the first impact ball, and generating a rule foul determination result; obtaining the terminal position information of the first impact ball, performing pocket identification, and generating a pocket identification result; generating the real-time scoring determination result according to the action foul determination result, the rule foul determination result and the pocket identification result.

[0050] Preferably, the position of the player's palm on the billiard table is extracted through image recognition technology, and the player's action detail features are extracted according to the palm coordinate area, including identifying the player's posture and batting action in the process of preparing to hit the ball and actually hitting the ball, for example, whether the cue is pushed with an improper posture or interferes with other balls. Based on the extracted player's action detail features, it is automatically determined whether the player has committed a foul, and a judgment is generated as to whether the player has committed an action foul, that is, an action foul judgment result is generated; then, according to the target impact ball (the billiard ball that collides with the white ball for the first time) and the player's ball distribution information (the batting order that the player must follow under the current game rules), a batting order rule is constructed. For example, in a snooker game, the player must first hit the red ball and then the colored ball, and the colored balls can be hit in order only after all the red balls have been hit; when the first impact ball is detected (that is, the first collision between the white ball and the target ball), it will be compared with the batting order. The system matches the order rules to ensure that the player hits the correct target ball. If the player violates the rules (for example, directly hitting the colored ball before hitting the red ball), it will be automatically judged as a foul, and then generate a rule foul judgment result; according to the collision between the white ball and the target ball, predict and obtain the end position of the first impacted ball, whether it enters the ball pocket, stays on the table, etc. Specifically, use computer vision and physical models to predict whether the ball enters the ball pocket along the correct trajectory, and judge whether the ball enters the ball pocket by calculating the end position of the target ball, and generate a pocket entry recognition result; finally, integrate the three results (action foul, rule foul, pocket entry situation) to generate the final real-time score judgment result, and feed back the final score judgment result to the game site in real time, including the current player's score, the opponent's player's score, and whether a foul occurred and the type of foul, thereby ensuring the intelligence and accuracy of billiards scoring.

[0051] In the above, refer to Figure 1 The intelligent billiard scoring device based on visual recognition according to an embodiment of the present invention is described in detail. Figure 2 The intelligent billiards scoring method based on visual recognition according to an embodiment of the present invention is described.

[0052] Intelligent billiard scoring method based on visual recognition, such as Figure 2 As shown, the method includes: using a high-performance image acquisition module to collect high-definition video stream data of a billiard table in real time; performing dual-channel parallel sampling processing on the high-definition video stream data to obtain a key frame detail image set and a motion capture image set; based on the key frame detail image set and the motion capture image set, performing rapid feature recognition to obtain multi-dimensional billiard motion features; after initializing a pre-trained intelligent billiard scoring model according to live billiard rules, using the initialized intelligent billiard scoring model to analyze and judge the multi-dimensional billiard motion features, determine the billiard motion state and perform score calculation, and generate a real-time score determination result.

[0053] In a possible implementation, the intelligent billiards scoring method based on visual recognition also includes: based on the high-definition video stream data, extracting high-definition desktop images and performing semantic segmentation, removing interfering background, and generating a desktop graphic collection; building a parallel sampling module, the parallel sampling module includes a detail capture channel and a motion capture channel; inputting the desktop graphic collection into the parallel sampling module, and the detail capture channel and the motion capture channel respectively perform key frame extraction and down sampling to obtain the key frame detail image set and the motion capture image set.

[0054] In a possible implementation, the intelligent billiards scoring method based on visual recognition also includes: extracting an original desktop image of a batting action cycle based on the key frame detail image set; performing target object recognition based on the original desktop image, calibrating multiple targets and target starting positions; and for the multiple targets, traversing the key frame detail image set and the motion capture image set in chronological order to extract multi-dimensional billiards motion features.

[0055] In a possible implementation, the intelligent billiards scoring method based on visual recognition also includes: for the multiple targets, traversing the motion capture image set in chronological order, and extracting the motion trajectory features of each target respectively; extracting trajectory mutation points based on the motion trajectory features, and matching key detail images from the key frame detail image set according to the time mark of the trajectory mutation point; extracting multidimensional detail features of the multiple targets based on the key detail images; and fusing the motion trajectory features and multidimensional detail features of each target to generate the multidimensional billiards motion features.

[0056] In a possible implementation, the smart billiards scoring method based on visual recognition also includes: collecting scoring rule data of various billiards game types, performing centralized analysis, and extracting common basic rule data as basic framework data; using the basic framework data to pre-build a smart billiards scoring model, the smart billiards scoring model includes a scoring determination unit and a foul detection unit; according to the on-site billiards rules, comparing and analyzing with the basic framework data, generating a rule adjustment vector, and adjusting the rule parameters of the smart billiards scoring model based on the rule adjustment vector.

[0057] In a possible implementation, the intelligent billiards scoring method based on visual recognition also includes: extracting multidimensional detail features of the original desktop image from the multidimensional billiards motion features based on the initialized intelligent billiards scoring model; extracting the target object area according to the multidimensional detail features of the original desktop image, including extracting the pocket coordinate area, the billiard ball coordinate area, the player's palm coordinate area, and the cue coordinate area; segmenting and calculating the target object area to generate a cue direction vector; based on the cue direction vector, combined with the player's ball distribution information, performing a hitting extension calculation to identify and obtain a target impact ball; extracting white ball motion trajectory data according to the multidimensional billiards motion features, and identifying and extracting a first impact ball therefrom; based on the basic information of the first impact ball, combining the target impact ball and the player's ball distribution information, performing a foul judgment and a score calculation to generate a real-time score judgment result.

[0058] In a possible implementation, the intelligent billiards scoring method based on visual recognition also includes: based on the player's palm coordinate area, extracting the player's action detail features, performing action foul judgment, and generating an action foul judgment result; generating a batting order rule according to the target impact ball and the player's ball distribution information, matching the information with the first impact ball, and generating a rule foul judgment result; obtaining the terminal position information of the first impact ball, performing pocket recognition, and generating a pocket recognition result; generating the real-time scoring judgment result according to the action foul judgment result, the rule foul judgment result and the pocket recognition result.

[0059] The intelligent billiards scoring device based on visual recognition provided by the embodiment of the present invention can execute the intelligent billiards scoring method based on visual recognition provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0060] Although the present application makes various references to certain modules in the device according to the embodiments of the present application, any number of different modules may be used and run on the user terminal and / or server, and the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0061] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. An intelligent billiard scoring device based on visual recognition, characterized in that: The device comprises: High-definition video stream data acquisition module, used to collect high-definition video stream data of the billiard table in real time through a high-performance image acquisition module; A parallel sampling processing module is used to perform dual-channel parallel sampling processing on the high-definition video stream data to obtain a key frame detail image set and a motion capture image set; A fast feature recognition module, used for performing fast feature recognition based on the key frame detail image set and the motion capture image set to obtain multi-dimensional billiards motion features; The real-time score determination result generation module is used to initialize the pre-trained intelligent billiards scoring model according to the live billiards rules, use the initialized intelligent billiards scoring model to analyze and judge the multi-dimensional billiards motion characteristics, determine the billiards motion state and perform score calculation, and generate a real-time score determination result.

2. The intelligent billiard scoring device based on visual recognition as claimed in claim 1, characterized in that: The parallel sampling processing module comprises: Based on the high-definition video stream data, extract the high-definition desktop image and perform semantic segmentation, remove the interfering background, and generate a desktop graphics collection; Building a parallel sampling module, wherein the parallel sampling module includes a detail capture channel and a dynamic capture channel; The desktop graphics collection is input into the parallel sampling module, and the detail capture channel and the motion capture channel respectively perform key frame extraction and down sampling to obtain the key frame detail image set and the motion capture image set.

3. The intelligent billiard scoring device based on visual recognition as claimed in claim 2, characterized in that: The rapid feature recognition module comprises: Extracting an original desktop image of a batting action cycle based on the key frame detail image set; Based on the original desktop image, target object recognition is performed, and multiple targets and starting positions of the targets are calibrated; For the multiple targets, the key frame detail image set and the motion capture image set are traversed in time order to extract multi-dimensional billiard motion features.

4. The intelligent billiard scoring device based on visual recognition as claimed in claim 3, characterized in that: The rapid feature recognition module comprises: For the multiple targets, traverse the motion capture image set in chronological order, and extract motion trajectory features of each target respectively; Extracting trajectory mutation points based on the motion trajectory features, and matching key detail images from the key frame detail image set according to time marks of the trajectory mutation points; Extracting multi-dimensional detail features of the multiple targets according to the key detail image; The motion trajectory features and multi-dimensional detail features of each target object are integrated to generate the multi-dimensional billiard motion features.

5. The intelligent billiard scoring device based on visual recognition as claimed in claim 4, characterized in that: The real-time score determination result generation module comprises: Collect scoring rule data of various types of billiard games, conduct centralized analysis, and extract common basic rule data as basic framework data; Using the basic framework data, pre-build an intelligent billiards scoring model, the intelligent billiards scoring model includes a score determination unit and a foul detection unit; According to the on-site billiard rules, a comparison analysis is performed with the basic framework data to generate a rule adjustment vector, and based on the rule adjustment vector, the rule parameters of the intelligent billiard scoring model are adjusted.

6. The intelligent billiard scoring device based on visual recognition as claimed in claim 5, characterized in that: The real-time score determination result generation module comprises: Based on the initialized intelligent billiards scoring model, extracting multi-dimensional detail features of the original desktop image from the multi-dimensional billiards motion features; According to the multi-dimensional detail features of the original desktop image, the target area is extracted, including the pocket coordinate area, billiard ball coordinate area, player palm coordinate area, and cue coordinate area; Segment and calculate the target area to generate a club direction vector; Based on the club direction vector and the player's ball distribution information, the hitting extension calculation is performed to identify and obtain the target impact ball; According to the multi-dimensional billiard motion characteristics, the motion trajectory data of the white ball is extracted, and the first impact ball is identified and extracted therefrom; According to the basic information of the first impact ball, the foul determination and score calculation are performed in combination with the target impact ball and the player's ball distribution information to generate a real-time score determination result.

7. The intelligent billiard scoring device based on visual recognition as claimed in claim 6, characterized in that: The real-time score determination result generation module comprises: Based on the palm coordinate area of ​​the player, extract the detailed features of the player's action, make an action foul determination, and generate an action foul determination result; Generate a batting order rule based on the target impact ball and the player's ball distribution information, match the information with the first impact ball, and generate a rule foul determination result; Acquire the terminal position information of the first impact ball, perform pocket identification, and generate a pocket identification result; The real-time score determination result is generated according to the action foul determination result, the rule foul determination result and the pocket identification result.

8. An intelligent billiard scoring method based on visual recognition, characterized in that: The method is applied to the intelligent billiards scoring device based on visual recognition according to any one of claims 1 to 7, and the method comprises: Through the high-performance image acquisition module, the high-definition video stream data of the billiard table is collected in real time; Performing dual-channel parallel sampling processing on the high-definition video stream data to obtain a key frame detail image set and a motion capture image set; Based on the key frame detail image set and the motion capture image set, fast feature recognition is performed to obtain multi-dimensional billiards motion features; After the pre-trained intelligent billiard scoring model is initialized according to the live billiard rules, the multi-dimensional billiard motion characteristics are analyzed and judged using the initialized intelligent billiard scoring model, the billiard motion state is determined and the score is calculated, and a real-time score determination result is generated.

Images