Video plane conversion method, device, electronic device and computer program product

Abstract

The present invention provides a video plane conversion method, device, electronic device and computer program product, the method comprising: determining the key points of the field according to the field model image; determining the coordinate position and confidence of the key points of the field in the video frame image based on the key point regression algorithm of the Gaussian heat map; determining the effective key points according to the sorting results of the confidence of the key points of the field in the video frame image and the principle of three points not being collinear, wherein the principle of three points not being collinear is used to indicate that the collinearity value of any three field key points is zero; determining the perspective transformation matrix according to the coordinate position of the effective key points in the field model image and the video frame image, the perspective transformation matrix is ​​used to realize the plane mapping of any position in the video frame image. The present invention can realize the plane conversion of the game video and intuitively express the global tactics from a bird's-eye view.

Description

Technical Field

[0001] The present invention relates to the technical field of video conversion, and in particular to a video plane conversion method, device, electronic equipment and computer program product. Background Art

[0002] At present, in sports competitions, such as football matches, when shooting videos, the football matches are often shot at a local offset angle based on the current position of the football. This is to reflect the characteristics of the event and avoid cluttered backgrounds, making it easier for the audience to watch the game from their perspective and to capture exciting and important moments of the game.

[0003] In the prior art, in videos shot at an offset angle, the area included in the video is often a small part of the court, highlighting the skills of some athletes, but not reflecting the entire court and the movements of other athletes, that is, the overall tactical system is not intuitive. Summary of the invention

[0004] The present invention provides a video plane conversion method, device, electronic device and computer program product, which are used to solve the defect that the overall tactical system is not intuitively expressed in the prior art, realize the plane conversion of the game video, and intuitively express the overall tactics from a bird's-eye view.

[0005] The present invention provides a video plane conversion method, comprising:

[0006] Determine key points of the site based on the site model image;

[0007] Based on the key point regression algorithm of Gaussian heat map, the coordinate position and confidence of the key point of the venue in the video frame image are determined;

[0008] Determine valid key points according to the ranking results of the confidences of the site key points in the video frame image and the principle that the three points are not collinear, wherein the principle that the three points are not collinear is used to indicate that the collinearity values ​​of any three of the site key points are zero;

[0009] A perspective transformation matrix is ​​determined according to the coordinate positions of the effective key points in the site model image and the video frame image, and the perspective transformation matrix is ​​used to achieve plane mapping of any position in the video frame image.

[0010] According to the video plane conversion method provided by the present invention, the Gaussian heat map-based key point regression algorithm determines the coordinate position and confidence of the venue key point in the video frame image, including:

[0011] Constructing a data set, wherein the data set includes a video frame image and a label Gaussian heat map produced based on the video frame image;

[0012] The data set based on data enhancement is input into a neural network to obtain a generated feature map, and the neural network is optimized using JS divergence as a loss function, wherein each generated feature map corresponds to a venue key point, that is, the coordinates of the corresponding venue key point are the coordinates of the maximum pixel point in each generated feature map;

[0013] The confidence level of the site key point is determined according to the predicted value of the corresponding coordinate of the site key point in the generated feature map.

[0014] According to the video plane conversion method provided by the present invention, the data set based on data enhancement is input into the neural network, including: the data set is input into the neural network after each random data enhancement, the data enhancement is used to overcome the position, perspective, and brightness offset of the key points of the scene in the data set, expand the data set and enhance the generalization of the neural network, wherein the data enhancement includes random translation within the enhancement threshold range, random degree rotation, random scaling, random transformation of color jitter parameters and random masking of pixels.

[0015] According to the video plane conversion method provided by the present invention, the valid key points are determined according to the sorting results of the confidences of the site key points in the video frame image and the principle that the three points are not collinear, wherein the principle that the three points are not collinear is used to indicate that the collinearity values ​​of any three of the site key points are zero, including:

[0016] Sorting the confidence levels of the key points of the venue in the video frame images;

[0017] Based on the confidence ranking result of the site key points, the site key points are traversed, and the collinearity values ​​of any three high-confidence site key points are used to obtain at least four valid key points whose three points are not collinear.

[0018] According to the video plane conversion method provided by the present invention, the confidence ranking result of the field key points is based on traversing the field key points, and using the collinearity values ​​of any three field key points with high confidence to obtain at least four valid key points that are not collinear, including:

[0019] Acquire two site key points with the highest confidence and the coordinates corresponding to the two site key points in the site model image, and use the two site key points with the highest confidence as valid key points;

[0020] Based on the confidence ranking result of the site key points, determine the collinearity value between the next site key point and the two site key points with the highest confidence in sequence; if the collinearity value is 1, discard the site key point, and determine the collinearity value between the next site key point and the two site key points with the highest confidence according to the ranking result; if the collinearity value is 0, retain the site key point and use it as a valid key point;

[0021] If the number of valid key points is greater than or equal to N, the traversal is terminated.

[0022] According to the video plane conversion method provided by the present invention, the perspective transformation matrix is ​​determined according to the coordinate positions of the effective key points in the site model image and the video frame image, and the perspective transformation matrix is ​​used to realize the plane mapping of any position in the video frame image, including:

[0023] Obtaining the coordinates of the valid key points in the site model image;

[0024] Obtaining the coordinates corresponding to the valid key points in the video frame image;

[0025] Determining mapping parameter values ​​based on coordinates of the valid key points in the site model image and the video frame image;

[0026] The perspective transformation matrix is ​​determined according to the mapping parameter value, and the perspective transformation matrix is ​​used to realize mapping of any position in the video frame image to the venue mapping plane, and the video frame image corresponds to the venue mapping plane one by one.

[0027] The present invention also provides a video plane conversion device, comprising:

[0028] A first determination module is used to determine the key points of the site according to the site model image;

[0029] A second determination module is used to determine the coordinate position and confidence of the site key point in the video frame image based on a key point regression algorithm of a Gaussian heat map;

[0030] A third determination module is used to determine valid key points according to the ranking results of the confidences of the site key points in the video frame image and the principle that three points are not collinear, wherein the principle that three points are not collinear is used to indicate that the collinearity values ​​of any three of the site key points are zero;

[0031] The fourth determination module is used to determine a perspective transformation matrix according to the coordinate positions of the effective key points in the site model image and the video frame image, wherein the perspective transformation matrix is ​​used to realize plane mapping of any position in the video frame image.

[0032] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned video plane conversion methods is implemented.

[0033] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the video plane conversion method described in any one of the above methods is implemented.

[0034] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the video plane conversion method described above is implemented.

[0035] The video plane conversion method, device, electronic device and computer program product provided by the present invention determine the field key points covering the entire field through modeling, and when the original field key points disappear, new key points can appear to replace them, thereby ensuring the functional feasibility of the present invention at the principle level; through the key point regression algorithm, based on the Gaussian heat map, the coordinate position and confidence of the scene key points in the video frame image are regressed, and the field key points are traversed through the confidence sorting, and based on the collinearity values ​​of the field key points, the valid key points are obtained, and according to the corresponding coordinate positions of the valid key points in the field model image and the video frame image, the perspective transformation matrix is ​​determined, and any position in the video frame image is mapped to the field mapping plane, so as to intuitively reflect the global tactics from a bird's-eye view. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is one of the flow charts of the video plane conversion method provided by the present invention;

[0038] Figure 2 It is a schematic diagram of the distribution of key points of the venue of the video plane conversion method provided by the present invention;

[0039] Figure 3 This is the second flow chart of the video plane conversion method provided by the present invention;

[0040] Figure 4 This is the third flow chart of the video plane conversion method provided by the present invention;

[0041] Figure 5 This is the fourth flow chart of the video plane conversion method provided by the present invention;

[0042] Figure 6 It is a structural schematic diagram of the video plane conversion device provided by the present invention;

[0043] Figure 7It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Combine the following Figure 1-Figure 5 The video plane switching method of the present invention is described.

[0046] Figure 1 FIG. 1 is one of the flow charts of the video plane conversion method provided by the present invention, such as Figure 1 As shown, the method includes:

[0047] Step 110: determining key points of the site according to the site model image.

[0048] Optionally, Figure 2 Schematic diagram of the distribution of key points of the video plane conversion method provided by the present invention. Figure 2 As shown in the figure, in order to achieve site coverage, the site model is obtained based on the site image modeling, and 33 site key points (p1, p2, ..., p 33 ), each site key point belongs to a category, and the site key point is marked in the video frame image, where each site key point corresponds to a measurable coordinate position (x i ,y i ), used to realize the position association between the venue model image and the video frame image. The selection of 33 venue key points realizes that when the original venue key points disappear in the video frame image, new key points can appear to replace them, ensuring the functional feasibility of the video plane conversion method of the present invention at the principle level.

[0049] Step 120, based on the key point regression algorithm of the Gaussian heat map, determine the coordinate position and confidence of the venue key point in the video frame image.

[0050] Optionally, Figure 3 FIG. 2 is a flow chart of the video plane conversion method provided by the present invention. Figure 3 As shown, the method includes:

[0051] Step 210: construct a data set, where the data set includes video frame images and label Gaussian heat maps made based on the video frame images.

[0052] Optionally, a certain number of football match videos are obtained, including but not limited to the Internet, and the original data set is obtained by extracting frames from the obtained football match videos. The video frame images in the original data set cover all key points of the field and are as balanced as possible.

[0053] Optionally, each video frame image is annotated with reference to the distribution of the key points of the venue. Each video frame image is annotated with 33 label Gaussian heat maps, which represent the labels of different categories of key points of the venue. The label Gaussian heat map and the data frame image together constitute a data set. In each label Gaussian heat map, the location of the key point of the venue is taken as the center and diffuses outward in the form of Gaussian distribution. The generation formula of the label Gaussian heat map is shown in formula (1):

[0054]

[0055] Among them, D(x,y) represents the distance from the key point of the site in the label Gaussian heat map, and Z(x,y) is the label Gaussian heat map.

[0056] Step 220, input the data set based on the data enhancement into the neural network, obtain the generated feature map, and optimize the neural network with JS divergence as the loss function, wherein each generated feature map corresponds to a venue key point, that is, the coordinates of the corresponding venue key point are the coordinates of the maximum pixel point in each generated feature map.

[0057] Optionally, the data set is input into the neural network after each random data enhancement, and the data enhancement is used to overcome the position, perspective, and brightness offset of key points of the site in the data set, expand the data set and enhance the generalization of the neural network, wherein the data enhancement includes random translation within the enhancement threshold range, random degree rotation, random scaling, random transformation of color jitter parameters, and random masking of pixels.

[0058] Optionally, data augmentation, including:

[0059] Random translation up and down, left and right within 100 pixels;

[0060] Random degree rotation within plus or minus 10 degrees;

[0061] Randomly scale the scale within 15%;

[0062] The color jitter parameters are randomly changed, including brightness, saturation, and contrast;

[0063] Random masking within 50 pixels. Random masking is used to simulate player interference.

[0064] Optionally, for each video frame image, 33 feature maps are generated by the neural network, that is, 33 generated feature maps Each generated feature map corresponds to a key point of the venue, and the coordinates of the point with the maximum brightness are the coordinates of the key point of the venue. The coordinate prediction formula of the kth key point of the venue is shown in formula (2):

[0065]

[0066] in, is the predicted coordinate of the kth site key point, Generate a feature map corresponding to the kth key point.

[0067] Optionally, the present invention uses JS divergence as a loss function to guide neural network optimization. Compared with KL divergence, JS divergence is used to solve the asymmetry when positive and negative samples are interchanged, and is a more effective indicator for measuring the distribution difference between generated feature maps and generated feature maps. In the optimization guidance, the difference between the generated feature map output by the neural network and the label Gaussian heat map is measured by JS divergence, and the difference value is back-propagated until the difference value becomes smaller and smaller, thereby providing an effective optimization basis and further guiding the neural network training. Among them, for the kth site key point, the formula of JS divergence is shown in formula (3):

[0068]

[0069] Among them, Z k and are the label Gaussian heat map and generated feature map of the k-th site key point, respectively. KL is the KL divergence, and the formula of KL divergence is shown in formula (4):

[0070]

[0071] Where i represents the coordinate point in the generated feature map, n represents the total number of coordinates in the image, and p is different in formula (3). KL where Z in formula (3) k and q corresponds to the

[0072] Step 230, based on the predicted values ​​of the corresponding coordinates of the site key points in the generated feature map, the confidence of the site key points is determined to provide a basis for the selection of effective key points required for the subsequent perspective transformation matrix calculation. The confidence generation formula is shown in formula (5):

[0073]

[0074] Step 130, determining valid key points according to the ranking results of the confidences of the venue key points in the video frame image and the three-point non-collinearity principle, wherein the three-point non-collinearity principle is used to indicate that the collinearity value of any three venue key points is zero.

[0075] Optionally, Figure 4 FIG. 3 is a flow chart of the video plane conversion method provided by the present invention. Figure 4 As shown, the method excludes invisible site key points by their confidence, filters the site key points by determining whether they belong to risk segments, obtains valid key points, and avoids the existence of three points in one line. The method includes:

[0076] Step 310: sort the confidences of the scene key points in the video frame images.

[0077] Step 320, based on the confidence ranking result of the site key points, traverse the site key points, and use the collinearity values ​​of any three high-confidence site key points to obtain at least four valid key points that are not collinear.

[0078] Optionally, the above method comprises:

[0079] The two site key points with the highest confidence and the corresponding coordinates of the two site key points in the site model image are obtained, and the two site key points with the highest confidence are used as valid key points;

[0080] Based on the confidence ranking results of the site key points, the collinearity values ​​of the next site key point and the two site key points with the highest confidence are determined in turn. If the collinearity value is 1, the site key point is discarded, and the collinearity value of the next site key point and the two site key points with the highest confidence is determined according to the ranking results. If the collinearity value is 0, the site key point is retained and used as a valid key point;

[0081] If the number of valid key points is greater than or equal to N, the traversal is terminated.

[0082] Optionally, obtain the coordinates of 33 site key points in the video frame image and its corresponding confidence of the key points of the site, and obtain the coordinates of the 33 key points of the site in the site model image p = [p1, p2, ..., p 33 ], and the coordinates of the kth site key point are p k =[x k ,y k ],in, Each element in p corresponds to a key point of the site.

[0083] Optionally, based on the confidence ranking result, two site key points with the highest confidence are selected and marked as (r1, r2). Based on the confidence ranking result, a third site key point is selected and marked as r3. The p in the site model image is calculated. r1 、p r2 、p r3 Are they collinear? r1 、p r2 、p r3 If there are three points on a line, then or or The slopes of any group are equal, that is, p r1 、p r2 、p r3 The collinearity value is 1, and the calculation formula of the collinearity value is shown in formula (6):

[0084]

[0085] in, is an indicator function. If the collinearity value is 1, then the p r3 Abandon it, and further select the next key point of the site based on the confidence ranking result, and re-mark it as p r3 ; If the collinearity value is 0, then p r3 Keep it and the two key points of the site with the highest confidence as valid key points, and further select p r4 The traversal stops when at least N valid key points are retained. For example, N is 4.

[0086] Step 140, determining a perspective transformation matrix according to the coordinate positions of the effective key points in the site model image and the video frame image, wherein the perspective transformation matrix is ​​used to realize plane mapping of any position in the video frame image.

[0087] Optionally, Figure 5 FIG. 4 is a flow chart of the video plane conversion method provided by the present invention. Figure 5 As shown, the method includes:

[0088] Get the coordinates of valid key points in the site model image.

[0089] Get the coordinates of valid key points in the video frame image.

[0090] Based on the coordinates of the valid key points in the site model image and the video frame image, the mapping parameter values ​​are determined.

[0091] According to the mapping parameter value, the perspective transformation matrix is ​​determined, and the perspective transformation matrix is ​​used to realize the mapping of any position in the video frame image to the venue mapping plane, and the video frame image and the venue mapping plane have a one-to-one correspondence.

[0092] Optionally, the mapping function between the site model image and the video frame image is set as shown in formula (7):

[0093]

[0094] Where [x,y,1] T is the coordinates of the key points of the venue in the video frame image, [x', y', z'] T is the coordinates of the key points of the site corresponding to the site model image, is the perspective transformation matrix, that is, the mapping relationship between the site model image and the video frame image. In the perspective transformation matrix, Used to realize arbitrary linear transformation of images, such as various flipping and scaling; [a 13 ,a 23 ] T Used to achieve image translation; [a 31 ,a 32 ] T Used to represent perspective transformation, a 33 Set to 1.

[0095] Since the video frame image refers to a three-dimensional space, a z-axis normalization operation is also required. The player position coordinates in the final field model image are shown in formula (8):

[0096]

[0097] A 33 =1 and substitute formula (8) into formula (7), and rearrange to obtain formula (9):

[0098]

[0099] Among them, the coordinates (x, y) of the four effective key points in the video frame image and the coordinates (x″, y″) of the four effective key points in the site model image are all known values, so formula (9) is linearly solvable. Substituting the four pairs of coordinates of the effective key points into formula (9), the mapping parameter values ​​(a 11 ~a 31 ), and then obtain the perspective transformation matrix to achieve the mapping of the video frame image to the site model image.

[0100] Optionally, after obtaining the perspective transformation matrix, video frame images are extracted frame by frame from the input video. The player coordinates in the video frame images can be obtained through target detection, target tracking and other technologies. The present invention is not limited to this. The player coordinate mapping in the venue model image is obtained through the player coordinates in the video frame image, and the venue model image mapped by the current video frame image is stored. The mapping process of the next video frame image is looped until the entire video processing is completed, and the video result is output.

[0101] The video plane conversion method provided by the present invention determines the field key points covering the entire field through modeling, and when the original field key points disappear, new key points can appear to replace them, thereby ensuring the functional feasibility of the present invention at the principle level; through the key point regression algorithm, based on the Gaussian heat map, the coordinate position and confidence of the scene key points in the video frame image are regressed, and the field key points are traversed through the confidence sorting, and based on the collinearity values ​​of the field key points, the valid key points are obtained, and according to the corresponding coordinate positions of the valid key points in the field model image and the video frame image, the perspective transformation matrix is ​​determined, and any position in the video frame image is mapped to the field mapping plane, so as to intuitively reflect the global tactics from a bird's-eye view.

[0102] The video plane conversion device provided by the present invention is described below. The video plane conversion device described below and the video plane conversion method described above can be referred to each other.

[0103] Figure 6 Schematic diagram of the structure of the video plane conversion device provided by the present invention. Figure 6 As shown, the video plane conversion device 400 includes a first determination module 401, a second determination module 402, a third determination module 403 and a fourth determination module 404, wherein:

[0104] A first determination module 401 is used to determine the key points of the site according to the site model image;

[0105] The second determination module 402 is used to determine the coordinate position and confidence of the venue key point in the video frame image based on the key point regression algorithm of the Gaussian heat map;

[0106] The third determination module 403 is used to determine the valid key points according to the ranking result of the confidence of the venue key points in the video frame image and the principle that the three points are not collinear, wherein the principle that the three points are not collinear is used to indicate that the collinearity value of any three venue key points is zero;

[0107] The fourth determination module 404 is used to determine the perspective transformation matrix according to the coordinate positions of the effective key points in the site model image and the video frame image, and the perspective transformation matrix is ​​used to achieve plane mapping of any position in the video frame image.

[0108] The video plane conversion device provided by the present invention determines the field key points covering the entire field through modeling. When the original field key points disappear, new key points can appear to replace them, thereby ensuring the functional feasibility of the present invention at the principle level; through the key point regression algorithm, based on the Gaussian heat map, the coordinate position and confidence of the scene key points in the video frame image are regressed, and the field key points are traversed through the confidence sorting, and based on the collinearity values ​​of the field key points, the valid key points are obtained, and according to the corresponding coordinate positions of the valid key points in the field model image and the video frame image, the perspective transformation matrix is ​​determined, and any position in the video frame image is mapped to the field mapping plane, so as to intuitively reflect the global tactics from a bird's-eye view.

[0109] Optionally, the first determination module 401 is specifically configured to select 33 site key points in the site model image based on considerations of full site coverage and point balance.

[0110] Optionally, the second determining module 402 is specifically configured to:

[0111] Construct a data set, which includes video frame images and label Gaussian heat maps made based on the video frame images;

[0112] Based on the data augmentation data set, the neural network is input to obtain the generated feature map, and the neural network is optimized using JS divergence as the loss function. Each generated feature map corresponds to a venue key point, that is, the coordinates of the corresponding venue key point are the coordinates of the maximum pixel point in each generated feature map.

[0113] The confidence of the site key points is determined based on the predicted values ​​of the corresponding coordinates of the site key points in the generated feature map.

[0114] Optionally, the second determining module 402 is specifically configured to:

[0115] The dataset is input into the neural network after each random data enhancement. Data enhancement is used to overcome the position, perspective, and brightness offset of key points of the site in the dataset, expand the dataset and enhance the generalization of the neural network. Data enhancement includes random translation within the enhancement threshold range, random degree rotation, random scaling, random transformation of color jitter parameters, and random masking of pixels.

[0116] Optionally, the third determining module 403 is specifically configured to:

[0117] Sort the confidence levels of the key points of the venue in the video frame images;

[0118] Based on the confidence ranking results of the site key points, the site key points are traversed, and the collinearity values ​​of any three high-confidence site key points are used to obtain at least four valid key points that are not collinear.

[0119] Optionally, the third determining module 403 is specifically configured to:

[0120] The two site key points with the highest confidence and the corresponding coordinates of the two site key points in the site model image are obtained, and the two site key points with the highest confidence are used as valid key points;

[0121] Based on the confidence ranking results of the site key points, the collinearity values ​​of the next site key point and the two site key points with the highest confidence are determined in turn. If the collinearity value is 1, the site key point is discarded, and the collinearity value of the next site key point and the two site key points with the highest confidence is determined according to the ranking results. If the collinearity value is 0, the site key point is retained and used as a valid key point;

[0122] If the number of valid key points is greater than or equal to N, the traversal is terminated.

[0123] Optionally, the fourth determining module 404 is specifically configured to:

[0124] Obtain the coordinates of valid key points in the site model image;

[0125] Get the coordinates of the valid key points in the video frame image;

[0126] Determine mapping parameter values ​​based on the coordinates of the valid key points in the site model image and the video frame image;

[0127] According to the mapping parameter value, the perspective transformation matrix is ​​determined, and the perspective transformation matrix is ​​used to realize the mapping of any position in the video frame image to the venue mapping plane, and the video frame image and the venue mapping plane have a one-to-one correspondence.

[0128] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device 500 may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the video plane switching method, which includes:

[0129] Determine key points of the site based on the site model image;

[0130] Based on the Gaussian heat map, the key point regression algorithm determines the coordinate position and confidence of the key points of the venue in the video frame image;

[0131] Determine the valid key points according to the confidence ranking results of the key points of the venue in the video frame image and the principle that the three points are not collinear, wherein the principle that the three points are not collinear is used to indicate that the collinearity value of any three key points of the venue is zero;

[0132] According to the coordinate positions of the effective key points in the site model image and the video frame image, the perspective transformation matrix is ​​determined, and the perspective transformation matrix is ​​used to realize the plane mapping of any position in the video frame image.

[0133] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0134] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the video plane conversion method provided by the above methods, the method includes:

[0135] Determine key points of the site based on the site model image;

[0136] Based on the Gaussian heat map, the key point regression algorithm determines the coordinate position and confidence of the key points of the venue in the video frame image;

[0137] Determine the valid key points according to the confidence ranking results of the key points of the venue in the video frame image and the principle that the three points are not collinear, wherein the principle that the three points are not collinear is used to indicate that the collinearity value of any three key points of the venue is zero;

[0138] According to the coordinate positions of the effective key points in the site model image and the video frame image, the perspective transformation matrix is ​​determined, and the perspective transformation matrix is ​​used to realize the plane mapping of any position in the video frame image.

[0139] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the video plane conversion method provided by the above methods is implemented, and the method includes:

[0140] Determine key points of the site based on the site model image;

[0141] Based on the Gaussian heat map, the key point regression algorithm determines the coordinate position and confidence of the key points of the venue in the video frame image;

[0142] Determine the valid key points according to the confidence ranking results of the key points of the venue in the video frame image and the principle that the three points are not collinear, wherein the principle that the three points are not collinear is used to indicate that the collinearity value of any three key points of the venue is zero;

[0143] According to the coordinate positions of the effective key points in the site model image and the video frame image, the perspective transformation matrix is ​​determined, and the perspective transformation matrix is ​​used to realize the plane mapping of any position in the video frame image.

[0144] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video plane conversion method, characterized in that: include: Determine key points of the site based on the site model image; Based on the key point regression algorithm of Gaussian heat map, the coordinate position and confidence of the key point of the venue in the video frame image are determined; Determine valid key points according to the ranking results of the confidences of the site key points in the video frame image and the principle that the three points are not collinear, wherein the principle that the three points are not collinear is used to indicate that the collinearity values ​​of any three of the site key points are zero; Determine a perspective transformation matrix according to the coordinate positions of the effective key points in the site model image and the video frame image, wherein the perspective transformation matrix is ​​used to realize plane mapping of any position in the video frame image; The determining of valid key points according to the ranking results of the confidences of the key points of the venue in the video frame image and the principle that the three points are not collinear includes: Sorting the confidence levels of the key points of the venue in the video frame images; Based on the confidence ranking result of the site key points, the site key points are traversed, and the collinearity values ​​of any three high-confidence site key points are used to obtain at least four valid key points that are not collinear.

2. The video plane conversion method according to claim 1, characterized in that: The Gaussian heat map-based key point regression algorithm determines the coordinate position and confidence of the site key point in the video frame image, including: Constructing a data set, wherein the data set includes a video frame image and a label Gaussian heat map produced based on the video frame image; The data set based on data enhancement is input into a neural network to obtain a generated feature map, and the neural network is optimized using JS divergence as a loss function, wherein each generated feature map corresponds to a venue key point, that is, the coordinates of the corresponding venue key point are the coordinates of the maximum pixel point in each generated feature map; The confidence level of the site key point is determined according to the predicted value of the corresponding coordinate of the site key point in the generated feature map.

3. The video plane conversion method according to claim 2, characterized in that: The data set input into the neural network based on data enhancement includes: the data set is input into the neural network after each random data enhancement, the data enhancement is used to overcome the position, perspective, and brightness offset of the key points of the site in the data set, expand the data set and enhance the generalization of the neural network, wherein the data enhancement includes random translation within the enhancement threshold range, random degree rotation, random scaling, random transformation of color jitter parameters and random masking of pixels.

4. The video plane conversion method according to claim 1, characterized in that: The confidence ranking result based on the site key points is traversed, and the collinearity values ​​of any three high-confidence site key points are used to obtain at least four valid key points that are not collinear, including: Acquire two site key points with the highest confidence and the coordinates corresponding to the two site key points in the site model image, and use the two site key points with the highest confidence as valid key points; Based on the confidence ranking result of the site key points, determine the collinearity value between the next site key point and the two site key points with the highest confidence in sequence; if the collinearity value is 1, discard the site key point, and determine the collinearity value between the next site key point and the two site key points with the highest confidence according to the ranking result; if the collinearity value is 0, retain the site key point and use it as a valid key point; If the number of valid key points is greater than or equal to N, the traversal is terminated.

5. The video plane conversion method according to claim 1, characterized in that: The determining of a perspective transformation matrix according to the coordinate positions of the effective key points in the site model image and the video frame image, wherein the perspective transformation matrix is ​​used to realize plane mapping at any position in the video frame image, includes: Obtaining the coordinates of the valid key points in the site model image; Obtaining the coordinates corresponding to the valid key points in the video frame image; Determining mapping parameter values ​​based on coordinates of the valid key points in the site model image and the video frame image; The perspective transformation matrix is ​​determined according to the mapping parameter value, and the perspective transformation matrix is ​​used to realize mapping of any position in the video frame image to the venue mapping plane, and the video frame image corresponds to the venue mapping plane one by one.

6. A video plane conversion device, characterized in that: include: A first determination module is used to determine the key points of the site according to the site model image; A second determination module is used to determine the coordinate position and confidence of the site key point in the video frame image based on a key point regression algorithm of a Gaussian heat map; A third determination module is used to determine valid key points according to the ranking results of the confidences of the site key points in the video frame image and the principle that three points are not collinear, wherein the principle that three points are not collinear is used to indicate that the collinearity values ​​of any three of the site key points are zero; A fourth determination module, used to determine a perspective transformation matrix according to the coordinate positions of the effective key points in the site model image and the video frame image, wherein the perspective transformation matrix is ​​used to realize plane mapping of any position in the video frame image; The third determination module is specifically used to: Sort the confidence levels of the key points of the venue in the video frame images; Based on the confidence ranking results of the site key points, the site key points are traversed, and the collinearity values ​​of any three high-confidence site key points are used to obtain at least four valid key points that are not collinear.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the video plane conversion method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video plane conversion method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video plane conversion method according to any one of claims 1 to 5 is implemented.

Images