Dual-vision video processing method and system

Abstract

The invention discloses a double-vision video processing method and system, and relates to the field of double-vision videos, and the method comprises the steps: obtaining an image on a billiard table through a binocular camera, carrying out the preprocessing of the image, carrying out the double-vision three-dimensional reconstruction, obtaining the movement direction and speed of a billiard ball, predicting a movement track, and carrying out the collision auxiliary judgment. According to the invention, through processing of the double-vision video and combination of billiard motion trail prediction, auxiliary judgment is provided for billiard collision, and a referee is helped to carry out judgment and cutting; moreover, in the process of processing the video image, the image is deblurred based on a dynamic position estimation model, so that the deblurring effect is enhanced; according to the method, stereo matching is optimized, and an energy function combining Census transformation and mutual information is adopted, so that the robustness under illumination change is improved, and the result accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of dual-vision video processing, and in particular to a dual-vision video processing method and system. Background Art

[0002] Billiards is a popular indoor sport in which a cue ball is used to hit the target ball into the pocket. The billiard collision rule is one of the core rules of billiards, which mainly involves the collision between the cue ball and the target ball or other balls.

[0003] Currently, billiards refereeing uses a combination of manual visual and video refereeing. Although the current video system can track the trajectory of the ball, since the system uses camera framing technology, ordinary cameras are limited by the speed and angle of acquisition and cannot fully monitor the collision between balls. Therefore, there are disputes in the judgment of whether some balls collide. Summary of the invention

[0004] In order to solve the technical problem of billiard ball collision judgment in the prior art, the present invention provides a dual-vision video processing method and system.

[0005] The present invention is achieved through the following technical solutions: A dual-vision video processing method for billiard ball collision monitoring comprises the following steps: S1: Video image acquisition; using a binocular camera to obtain images on the billiard table; S2: Image preprocessing, including denoising, image segmentation, and target extraction; S3: Dual-vision 3D reconstruction; including: S31: Camera calibration, using the calibration board to take multiple sets of images and calculate the camera parameters through the calibration algorithm; S32: Image correction, stereo matching; S33: matching model parameter optimization; S34: Depth calculation, point cloud generation; S35: 3D reconstruction and visualization; S4: Obtain the moving direction and speed of the billiard ball according to the three-dimensional reconstructed image and the frame number of the camera; S5: Predict the motion trajectory, perform trajectory matching verification, and then perform collision auxiliary judgment.

[0006] Furthermore, the step S2 comprises: S21: image denoising and clarity processing; including denoising based on non-local mean, establishing a dynamic position estimation model for estimating the dynamic position offset and deblurring the image based on the dynamic position offset; S22: image segmentation, obtaining a partial image of the billiard table top; S23: Target extraction, obtaining the ball outline on the billiard table and locating the center point of the ball.

[0007] Furthermore, the dynamic position estimation model includes two neural networks, a first network and a second network, wherein the first network is used to preliminarily estimate the offset of the dynamic position, and the second network is used to accurately correct the dynamic position offset; the loss function of the dynamic position estimation model training process is as follows:

[0008] in, They are regularization loss, cross entropy loss, total variation loss, and mean square error loss, respectively. is the corresponding weight; The deblurring process uses a self-supervised convolutional neural network model, and the loss function of the model is as follows:

[0009] in, is the multi-scale content loss function, is the multi-scale frequency reconstruction loss function, is the weight coefficient.

[0010] Furthermore, the step S3 includes a stereo matching process, which uses a semi-global algorithm to perform stereo matching, and combines Census transformation and mutual information to determine a matching cost function.

[0011] Furthermore, the matching cost function is expressed as follows:

[0012] in, , are the control parameters of Census transformation and mutual information, is the cost function based on Census transformation, is the cost function based on mutual information, is the standardization function.

[0013] Furthermore, the matching model parameter optimization includes the following steps: a. Randomly select n samples from the data set P as the initial subset to initialize the stereo matching model; b. Set an error threshold, and select samples whose error with model M is less than the threshold from the residual set of data set P, record it as set S, and merge S with the initial subset to form set C; c. Based on set C, re-estimate stereo matching parameters and update stereo matching; repeat the above steps to iteratively optimize the model; d. When the sampling times reaches the preset value, the model with the largest internal point set C is selected from all generated models as the final result.

[0014] Furthermore, the step S5 comprises: S51: predicting the movement trajectory according to the obtained movement direction and speed of the ball; S52: Calculating the matching degree between the predicted motion trajectory and the actual motion trajectory, where the matching degree includes the trajectory point matching degree and the trajectory length matching degree; S53: Perform auxiliary collision judgment according to the matching degree verification result; By comparing the matching verification result with the set threshold, it is determined whether the target ball has collided or not.

[0015] Furthermore, step S51 includes: based on the LSTM network model, the movement direction, speed and position of all balls in the previous n moments are input into the LSTM network model, and the movement direction, speed and position of the balls in the next m moments are output until the speeds of all balls are 0, thereby forming the movement trajectory prediction results of all balls.

[0016] The present invention also provides a dual-vision video processing system, based on the dual-vision video processing method as described above, comprising: A video image acquisition module, which is used to obtain images on the billiard table using a binocular camera; Image preprocessing module, which is used to remove noise, clarify images, segment images, and extract targets; A dual-vision 3D reconstruction module, which is used to perform 3D reconstruction of image information collected by the binocular camera; A billiards motion parameter acquisition module, which is used to obtain the motion direction and speed of the billiards; The collision auxiliary judgment module is used to predict the motion trajectory, perform trajectory matching verification, and then perform collision auxiliary judgment.

[0017] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which program instructions of the dual-vision video processing method are stored, and the program instructions of the dual-vision video processing method can be executed by one or more processors to implement the steps of the dual-vision video processing method as described above.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention realizes the optimization processing of dual-vision video, and combines the prediction of billiard ball motion trajectory to provide auxiliary judgment for billiard ball collision and help referees make judgments; and in the process of processing video images, the present invention uses a dynamic position estimation model to deblur the image, thereby enhancing the deblurring effect; the present invention also optimizes stereo matching, uses an energy function combining Census transformation and mutual information, improves the robustness under illumination changes, and improves the accuracy of the results. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 It is a flowchart of a dual-vision video processing method according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0021] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0022] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The illustrations only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0023] like Figure 1 As shown, a dual-vision video processing method includes the following steps: S1: Video image acquisition; using a binocular camera to obtain images on the billiard table; S2: Image preprocessing, including denoising, image segmentation, and target extraction; The specific steps include: S21: Image denoising and clarity processing; the camera generates inevitable noise due to factors such as exposure environment, so the image needs to be denoised; since the billiards movement process will cause motion blur in the image, clarity processing is also required; including the following steps: a. Based on non-local mean denoising, the calculation steps are as follows:

[0024] Among them, N () is the non-local mean denoising function, q is the image before processing, and u is the denoised image. is a similarity coefficient, indicating the similarity between pixel points i and j. Optionally, the similarity is obtained by geometric distance. The above denoising algorithm utilizes the redundant information in the image and can well preserve the texture details in the image while removing the noise. It has the advantages of simplicity, good performance and easy improvement.

[0025] Optionally, in order to further improve the denoising effect, the present invention uses wavelet transform to perform multi-scale decomposition on the image, and then applies non-local mean denoising to each scale. Specifically, multi-level wavelet decomposition is performed on the noisy image, and non-local mean denoising is applied to each scale after decomposition, and finally the image is reconstructed. The multi-level wavelet threshold processing method can effectively eliminate high-frequency noise in the noisy image.

[0026] b. Establish a dynamic position estimation model to estimate the dynamic position offset; The dynamic position estimation model includes two neural networks, a first network and a second network, wherein the first network is used to preliminarily estimate the offset of the dynamic position, and the second network is used to accurately correct the offset of the dynamic position; Specifically, the first network is a cross-attention neural network, which adaptively learns important features of different channels and spaces to estimate the dynamic position information of the dynamic blurred image; the second network is a BP neural network.

[0027] The training process of the network is to give a blurred image in the training set and a corresponding clear image. The dynamic position estimation model takes the blurred image as input and outputs the dynamic position offset of the clear image, including horizontal and vertical directions.

[0028] Optionally, the loss function of the model training process is as follows:

[0029] in, They are regularization loss, cross entropy loss, total variation loss, and mean square error loss, respectively. is the corresponding weight.

[0030] c. Deblurring the image based on the dynamic position offset; The deblurring process uses a self-supervised convolutional neural network model, and the loss function of the model is as follows:

[0031] in, is the multi-scale content loss function, is the multi-scale frequency reconstruction loss function, is the weight coefficient.

[0032] S22: image segmentation, obtaining a partial image of the billiard table top; Optionally, the preprocessed image is segmented based on a watershed image segmentation algorithm to remove the background and obtain a partial image of the billiard table top.

[0033] S23: Target extraction, obtaining the ball outline on the billiard table and locating the center point of the ball.

[0034] S3: Dual-vision 3D reconstruction, including the following steps: S31: Camera calibration: Use the calibration board to take multiple sets of images and calculate the camera parameters through the calibration algorithm.

[0035] S32: Image correction, stereo matching; Use the calibration results to calculate the correction map and apply the correction transformation to the left and right images; find corresponding points in the left and right images, calculate the disparity, and obtain the disparity map; Optionally, the stereo matching adopts a semi-global algorithm (Semi-Global Matching), including matching cost calculation, matching cost aggregation, disparity calculation, and disparity post-processing.

[0036] Furthermore, the present invention adopts the following energy function:

[0037] Among them, r is the direction of cost traversal calculation, n is all directions, It represents the matching cost in each direction when the disparity value at point p is d, which is specifically expressed as:

[0038] in, is the matching cost function with a depth of d at point p, The cost aggregation is traversed in direction r at a depth of d at point p. The variable cost aggregation of the pixel before point p in the direction r with a depth of d is: The variable cost aggregation of the pixel before point p in the r direction at depth d+1, The variable cost aggregation of the pixel before point p in the r direction at a depth of d-1, is the variable cost aggregation of the pixel before point p in the direction r with depth i, The minimum value of the variable cost aggregation of the pixel before point p in the r direction; , are the penalty factors for smoothness constraint and edge constraint respectively, k and i are variable parameters.

[0039] The present invention also combines Census transformation and mutual information to determine the matching cost function, which is expressed as follows:

[0040] in, , are the control parameters of Census transformation and mutual information, is the cost function based on Census transformation, is the cost function based on mutual information, is the standardized function, the expression is as follows:

[0041] When c and When they are all positive numbers, The function value range is [0,1].

[0042]

[0043] is the Hamming distance with depth d, , They are two bit strings after census transformation.

[0044]

[0045]

[0046] in, is the cost of pixel p for disparity d, is the gray value of point p in the reference image b and point q in the matching image m after correction by the disparity image D, is the unit of grayscale mutual information; , is the grayscale distribution of the image, is the grayscale joint entropy.

[0047] The present invention uses grayscale mutual information as the cost metric for image matching. The method analyzes pixel grayscale value data, extracts matching cost values ​​from grayscale entropy lists and grayscale joint entropy distributions, and converts macroscopic parameters into effective evaluation indicators for pixel-level data, thereby significantly improving matching accuracy.

[0048] The present invention adopts an energy function combining Census transformation and mutual information to improve the robustness under illumination changes.

[0049] S33: Matching model parameter optimization

[0050] Since there may be abnormal data in stereo matching, in order to improve the accuracy of the results, the present invention performs matching optimization based on the following steps: a. Randomly select n samples from the dataset P as the initial subset to initialize the stereo matching model.

[0051] b. Set the error threshold, and select the samples whose error with model M is less than the threshold from the residual set of data set P, and record it as set S. Merge S with the initial subset to form set C.

[0052] c. Re-estimate stereo matching parameters based on set C and update stereo matching. Repeat the above steps to iteratively optimize the model.

[0053] d. When the sampling times reaches the preset value, the model with the largest internal point set C is selected from all generated models as the final result.

[0054] S34: Depth calculation, point cloud generation; The depth value of each pixel is calculated based on the disparity map and the depth map is converted into a 3D point cloud.

[0055] S35: 3D reconstruction and visualization

[0056] Convert point clouds into 3D models and visualize them.

[0057] Optionally, use surface reconstruction algorithms such as Poisson reconstruction and Delaunay triangulation to generate a mesh model and use image processing tools for visualization.

[0058] S4: Get the direction and speed of the billiard ball

[0059] According to the three-dimensional reconstructed image and the frame number of the camera, the moving direction and speed of the billiard ball are obtained; the steps include: a. Initialize parameters, and perform brightness equalization and normalization on the image; b. Construct the loss function and optimize the YOLO network parameters through back propagation to perform network training; c. Input the 3D reconstructed image into the YOLO network convolution layer to obtain the features of each frame and classify the feature information; d. Obtain the direction and speed of the billiard ball in each frame based on the network output.

[0060] S5: Predict the motion trajectory, perform trajectory matching verification, and then perform collision auxiliary judgment; S51: predicting the motion trajectory according to the obtained motion direction and speed; Specifically, based on the LSTM network model, the movement direction, speed and position of all balls in the previous n moments are input into the LSTM network model, and the movement direction, speed and position of the balls in the next m moments are output until the speed of all balls is 0, thereby forming the movement trajectory prediction results of all balls.

[0061] S52: Calculate the matching degree between the predicted motion trajectory and the actual motion trajectory; specifically, the matching degree is calculated according to the following formula, which includes the matching degree of the number of trajectory points: And the track length matching :

[0062] Among them, the number of matched trajectory points refers to the number of trajectory points that match the predicted motion trajectory and the actual motion trajectory; the total number of trajectory points in the entire path refers to the total number of points in the actual motion trajectory.

[0063]

[0064] S53: Perform auxiliary collision judgment according to the matching degree verification result; By comparing the matching verification result with the set threshold, it is determined whether the target ball has collided or not.

[0065] In this embodiment, dual-vision video processing is achieved, combined with billiard ball motion trajectory prediction, to provide auxiliary judgment for billiard ball collision and help the referee make a judgment; and, in the process of processing video images, the present invention uses a dynamic position estimation model to deblur the image, thereby enhancing the deblurring effect; the present invention also optimizes stereo matching, using an energy function that combines Census transformation and mutual information, thereby improving the robustness under illumination changes and improving the accuracy of the results.

[0066] The embodiment of the present invention further provides a dual-vision video processing system, based on the dual-vision video processing method as described above, comprising: A video image acquisition module, which is used to obtain images on the billiard table using a binocular camera; Image preprocessing module, which is used to remove noise, clarify images, segment images, and extract targets; A dual-vision 3D reconstruction module, which is used to perform 3D reconstruction of image information collected by the binocular camera; A billiards motion parameter acquisition module, which is used to obtain the motion direction and speed of the billiards; The collision auxiliary judgment module is used to predict the motion trajectory, perform trajectory matching verification, and then perform collision auxiliary judgment.

[0067] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which program instructions of a dual-vision video processing method are stored. The program instructions of the dual-vision video processing method can be executed by one or more processors to implement the steps of the dual-vision video processing method as described above.

[0068] The embodiments described above are only descriptions of the preferred implementation modes of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A dual-vision video processing method, characterized in that: For billiards video processing, including the following steps: S1: Video image acquisition; using a binocular camera to obtain images on the billiard table; S2: Image preprocessing; Including denoising and clarity, image segmentation, and target extraction; S3: Dual-vision 3D reconstruction; including: S31: Camera calibration, using the calibration board to take multiple sets of images and calculate the camera parameters through the calibration algorithm; S32: Image correction, stereo matching; S33: matching model parameter optimization; S34: Depth calculation, point cloud generation; S35: 3D reconstruction and visualization; S4: Obtain the moving direction and speed of the billiard ball according to the three-dimensional reconstructed image and the frame number of the camera; S5: Predict the motion trajectory, perform trajectory matching verification, and then perform collision auxiliary judgment.

2. The dual vision video processing method according to claim 1, characterized in that: The step S2 comprises: S21: image denoising and clarity processing; including denoising based on non-local mean, establishing a dynamic position estimation model for estimating the dynamic position offset and deblurring the image based on the dynamic position offset; S22: image segmentation, obtaining a partial image of the billiard table top; S23: Target extraction, obtaining the ball outline on the billiard table and locating the center point of the ball.

3. The dual vision video processing method according to claim 2, characterized in that: The dynamic position estimation model includes two neural networks, a first network and a second network. The first network is used to preliminarily estimate the dynamic position offset, and the second network is used to accurately correct the dynamic position offset. The loss function of the dynamic position estimation model training process is as follows: in, They are regularization loss, cross entropy loss, total variation loss, and mean square error loss, respectively. is the corresponding weight; The deblurring process uses a self-supervised convolutional neural network model, and the loss function of the model is as follows: in, is the multi-scale content loss function, is the multi-scale frequency reconstruction loss function, is the weight coefficient.

4. The dual vision video processing method according to claim 1, characterized in that: The step S3 includes a stereo matching process, which uses a semi-global algorithm to perform stereo matching and combines Census transformation and mutual information to determine a matching cost function.

5. The dual vision video processing method according to claim 4, characterized in that: The matching cost function is expressed as follows: in, , are the control parameters of Census transformation and mutual information, is the cost function based on Census transformation, is the cost function based on mutual information, is the standardization function.

6. The dual vision video processing method according to claim 1, characterized in that: The matching model parameter optimization comprises the following steps: a. Randomly select n samples from the data set P as the initial subset to initialize the stereo matching model; b. Set an error threshold, and select samples whose error with model M is less than the threshold from the residual set of data set P, record it as set S, and merge S with the initial subset to form set C; c. Based on set C, re-estimate stereo matching parameters and update stereo matching; repeat the above steps to iteratively optimize the model; d. When the sampling times reaches the preset value, the model with the largest internal point set C is selected from all generated models as the final result.

7. The dual vision video processing method according to claim 1, characterized in that: The step S5 comprises: S51: predicting the movement trajectory according to the obtained movement direction and speed of the ball; S52: Calculating the matching degree between the predicted motion trajectory and the actual motion trajectory, where the matching degree includes the trajectory point matching degree and the trajectory length matching degree; S53: Perform auxiliary collision judgment according to the matching degree verification result; By comparing the matching verification result with the set threshold, it is determined whether the target ball has collided or not.

8. The dual vision video processing method according to claim 7, characterized in that: The step S51 includes: based on the LSTM network model, the movement direction, speed and position of all balls in the previous n moments are input into the LSTM network model, and the movement direction, speed and position of the balls in the next m moments are output until the speed of all balls is 0, thereby forming the movement trajectory prediction results of all balls.

9. A dual-vision video processing system, based on the dual-vision video processing method according to any one of claims 1 to 8, comprising: A video image acquisition module, which is used to obtain images on the billiard table using a binocular camera; Image preprocessing module, which is used to remove noise, clarify images, segment images, and extract targets; A dual-vision 3D reconstruction module, which is used to perform 3D reconstruction of image information collected by the binocular camera; A billiards motion parameter acquisition module, which is used to obtain the motion direction and speed of the billiards; The collision auxiliary judgment module is used to predict the motion trajectory, perform trajectory matching verification, and then perform collision auxiliary judgment.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions of the dual-vision video processing method, and the program instructions of the dual-vision video processing method can be executed by one or more processors to implement the steps of the dual-vision video processing method as described in any one of claims 1 to 8.

Images