A video type intelligent recognition method based on a VR device

Abstract

This invention discloses a video type intelligent recognition method based on VR devices, comprising the following steps: S1, performing policy recognition on the video file and obtaining the policy recognition result; S2, performing model recognition on the video file and obtaining the model recognition result; S3, determining whether the policy recognition result and the model recognition result are the same. If they are the same, the policy recognition result is used as the final recognition result; if they are different, the model recognition result is used as the final recognition result. The advantages are: it can quickly, efficiently, and accurately identify VR video types, and it facilitates fast and accurate video rendering by video players.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for intelligent video type recognition based on VR devices. Background Technology

[0002] In VR video playback scenarios, there are various types of videos, such as 180-degree 3D, 360-degree 3D, and YTB 3D. VR devices need to know which type of video they belong to in order to correctly render video images that conform to human eye viewing habits. However, most of the existing recognition technologies on the market are designed for 180-degree, 360-degree, and 2D video recognition, and cannot recognize YTB, 360-degree 3D, and other video types. The existing recognition technologies cannot meet the recognition needs of all existing video types. Summary of the Invention

[0003] The purpose of this invention is to provide a video type intelligent recognition method based on VR devices, thereby solving the aforementioned problems existing in the prior art.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A method for intelligent video type recognition based on VR devices includes the following steps:

[0006] S1. Perform policy recognition on the video file and obtain the policy recognition results; S1 includes,

[0007] S11. Preprocess the video file: Remove the intro and outro of the video file, extract two keyframes, and save them as the first keyframe image.

[0008] S12. Determine whether the first keyframe image is a video with a top-bottom structure; if yes, proceed to step S13; if no, proceed to step S14.

[0009] S13. Sequentially determine whether the first keyframe image is a 360-degree video or a 180-degree video. If it is a 360-degree video, return that the video file is a 360-degree video in 3D. If it is a 180-degree video, return that the video file is a 180-degree video in 3D. If neither is true, return that the video file is a normal video in 3D.

[0010] S14. Determine whether the first keyframe image is a video with a left-right structure; if yes, proceed to step S15; if no, proceed to step S16.

[0011] S15. Sequentially determine whether the first keyframe image is a 3D ytb video, a 360-degree video, or a 180-degree video. If it is a 3D ytb video, return that the video file is a 3D ytb video; if it is a 360-degree video, return that the video file is a 360-degree video (left and right 3D); if it is a 180-degree video, return that the video file is a 180-degree video (left and right 3D); if it is none of the three, return that the video file is a normal video (left and right 3D).

[0012] S16. Sequentially determine whether the first keyframe image is a YTB video, a 360-degree video, or a 180-degree video. If it is a YTB video, return that the video file is a YTB video; if it is a 360-degree video, return that the video file is a 360-degree video; if it is a 180-degree video, return that the video file is a 180-degree video; if it is none of the three, return that the video file is a normal video.

[0013] S2. Perform model recognition on the video file and obtain the model recognition results; S2 includes,

[0014] S21. Preprocess the video file: After removing the beginning and end of the video file, extract a key frame and save it as the second key frame image.

[0015] S22. Preprocess the second keyframe image;

[0016] S23. Input the preprocessed second keyframe image into the Swin transformer model for image classification and return the probability value of each category. The category with the highest probability value is the category to which the video file belongs.

[0017] S3. Determine whether the policy recognition result and the model recognition result are the same. If they are the same, the policy recognition result shall be used as the final recognition result; if they are different, the model recognition result shall be used as the final recognition result.

[0018] Preferably, step S12 specifically involves: the first keyframe image has a height of h and a width of w; the first keyframe image is cut at 0.5h, resulting in two images, an upper image and a lower image, each with a height of 0.5h and a width of w; the ORB feature point matching + mean hash algorithm is used to calculate the similarity between the cut upper and lower images; if the similarity calculation result is greater than the upper and lower structure threshold, the two images are considered similar, the video file is determined to be an upper and lower structure, the cut upper image is saved, and the process proceeds to step S13; otherwise, the process proceeds to step S14.

[0019] Preferably, step S13 specifically includes the following:

[0020] S131. Extract the leftmost 5 columns of pixels from the top image and save them as Image 1, and the rightmost 5 columns of pixels and save them as Image 2. Convert Image 1 and Image 2 into two 64-dimensional vectors. Calculate the similarity between these two vectors using a cosine similarity + mean hash algorithm. If the similarity is greater than the 360-degree video vertical 3D threshold, the two images are considered similar, and the video file is determined to be a 360-degree video vertical 3D. Otherwise, proceed to step S132.

[0021] S132. If the aspect ratio of the upper image is equal to 1 or 2, the video file is determined to be a 180-degree vertical 3D video; otherwise, the video file is determined to be a normal vertical 3D video.

[0022] Preferably, step S14 specifically involves: the first keyframe image has a height of h and a width of w; the first keyframe image is cut at 0.5w, resulting in a left image and a right image with a height of h and a width of 0.5w; the ORB feature point matching + mean hash algorithm is used to calculate the similarity between the cut left and right images; if the similarity calculation result is greater than the left-right structure threshold, it indicates that the two images are similar, the video file is determined to be a left-right structure, and the cut left image is saved.

[0023] Preferably, step S15 specifically includes the following:

[0024] S151. Project the left image onto a six-sided cube. Take two columns of pixels at the junction of two adjacent faces of the six-sided cube. Convert the two columns of pixels into two 64-dimensional vectors. Calculate the cosine similarity between the two vectors. If the cosine similarity calculation result is greater than the ytb video 3D threshold, then determine that the video file is a ytb video 3D; otherwise, proceed to step S152.

[0025] S152. Extract the leftmost 5 columns of pixels from the left image and save them as image 3, and the rightmost 5 columns of pixels and save them as image 4. Convert image 3 and image 4 into two 64-dimensional vectors. Calculate the similarity between these two vectors using a cosine similarity + mean hash algorithm. If the similarity is greater than the 360-degree video left and right 3D threshold, the two images are considered similar, and the video file is determined to be a 360-degree video left and right 3D. Otherwise, proceed to step S153.

[0026] S153. If the aspect ratio of the left image is equal to 1 or 2, the video file is determined to be a 180-degree horizontal 3D video; otherwise, the video file is determined to be a normal horizontal 3D video.

[0027] Preferably, step S16 specifically includes the following steps:

[0028] S161. Project the first keyframe image into a six-sided cube. Take two columns of pixels at the junction of two adjacent faces of the six-sided cube. Convert the two columns of pixels into two 64-dimensional vectors. Calculate the cosine similarity between the two vectors. If the cosine similarity calculation result is greater than the ytb video threshold, then determine that the video file is a ytb video; otherwise, proceed to step S162.

[0029] S162. Extract the leftmost 5 columns of pixels from the first keyframe image and save it as image 5, and the rightmost 5 columns of pixels and save it as image 6. Convert image 5 and image 6 into two 64-dimensional vectors. Calculate the similarity between the two vectors using the cosine similarity + mean hash algorithm. If the similarity is greater than the 360-degree video threshold, it means that the two images are similar, and the video file is determined to be a 360-degree video; otherwise, proceed to step S163.

[0030] S163. If the aspect ratio of the first keyframe image is equal to 1 or 2, then the video file is determined to be a 180-degree video; otherwise, the video file is determined to be a normal video.

[0031] The preferred ORB feature point matching + mean hashing algorithm is as follows:

[0032] Key points are extracted from the two images obtained from the segmentation. The key points extracted from the two images are compared and matched to calculate the matching degree. The two images are then subjected to mean hashing to calculate the Hamming distance. If the calculated matching degree is greater than the matching threshold and the Hamming distance is less than the distance threshold, it means that the two images are similar.

[0033] The preferred cosine similarity + mean hash algorithm is as follows:

[0034] The possible values ​​of each of the three primary colors, ranging from 0 to 255, are divided into four partitions: [0–63], [64–127], [128–191], and [191–255]. The partition corresponding to each primary color is calculated, and this partition value is mapped to an integer between 0 and 63. The indices of the left and right pixels are calculated, and the number of pixels corresponding to each index is summed to obtain a 64-dimensional vector for the image. Cosine similarity is calculated for the 64-dimensional vectors of the two images obtained from the segmentation. Then, mean hashing is performed on the two images to calculate the Hamming distance. If the cosine similarity is greater than the cosine similarity threshold and the Hamming distance is less than the distance threshold, the two images are considered similar.

[0035] Preferably, step S22 specifically includes the following:

[0036] S221, the transforms.Resize function crops the image of the second keyframe;

[0037] S222. Use the Cutout and Mixup functions to perform data augmentation on the cropped image;

[0038] S223. Use the ToTensor function to tensorize the data-augmented image;

[0039] S224. Use the transforms.Normalize function to normalize the tensor image.

[0040] Preferably, step S23 specifically includes the following:

[0041] S231. The height of the second keyframe image is H and the width is W. The preprocessed image data is input into the PatchMerging layer. The patch partition divides the input image HxWx3 into a set of non-overlapping patches, where each patch is 4x4 in size, the feature dimension of each patch is 4x4x3=48, and the number of patch blocks is H / 4xW / 4.

[0042] S232. Input the image data output from the Patch Merging layer into the LineEmbedding and SwintransformerBlock layers. The linear embedding layer transforms the feature dimension of the divided patch into C, and then sends it into the Swin Transformer Block layer. Through a patch merging, the input is merged into adjacent patches in a 2x2 pattern, making the number of patch blocks H / 8xW / 8 and the feature dimension 4C.

[0043] S233. The image data output from the LineEmbedding and SwintransformerBlock layers is fed into the PatchMerging and SwintransformerBlock layers. The PatchMerging and SwintransformerBlock layers perform fine-grained image feature extraction through a self-attention mechanism. The self-attention mechanism calculates self-attention based on windows, with each window having a default of 7*7 patches. The self-attention calculation is completed within these windows. During the self-attention calculation process, the complexity is... The calculation formula is as follows:

[0044]

[0045] Where h, ω, and C′ represent the height, width, and number of channels of the feature map, respectively;

[0046] S234. Input the image data output from the Patch Merging and SwintransformerBlock layers into the MLP layer, and calculate the probability value of each category using the Softmax function. The category with the highest probability value is the category to which the video file belongs. The formula for the Softmax function is as follows:

[0047]

[0048] Among them, z i is the output value of the i-th category after the image data is processed by Patch Merging and SwintransformerBlock layers, and K is the number of categories.

[0049] The beneficial effects of this invention are: the method of this invention uses computer vision and image correlation technology to identify the type of video, and the VR video player can directly render the correct video image after obtaining the video type. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the strategy identification process in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the model recognition process in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the structure of the recognition model in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0054] In this embodiment, the mainstream VR video types currently include: ordinary video, ordinary video top-bottom 3D, ordinary video left-right 3D, 180-degree video, 180-degree video top-bottom 3D, 180-degree video left-right 3D, 360-degree video, 360-degree video top-bottom 3D, 360-degree video left-right 3D, ytb video, and ytb video 3D. This invention provides an intelligent video type recognition method based on VR devices, identifying 11 mainstream VR video types. First, the VR device itself identifies the video, mainly based on the image features of video screenshots for analysis and calculation; this is referred to as strategy recognition. Then, the strategy recognition results are verified. Verification involves training a large-scale image dataset to obtain a prediction model, which is then used to determine the video type. The method of this invention includes the following steps:

[0055] S1. Perform policy recognition on the video file and obtain the policy recognition results;

[0056] S2. Perform model recognition on the video file and obtain the model recognition results; S2 includes,

[0057] S3. Determine whether the policy recognition result and the model recognition result are the same. If they are the same, the policy recognition result shall be used as the final recognition result; if they are different, the model recognition result shall be used as the final recognition result.

[0058] The method of this invention mainly includes three parts: strategy identification, model identification, and comprehensive strategy identification, as well as a comprehensive judgment based on the results of model identification. These three parts will be explained in detail below.

[0059] I. Strategy Identification

[0060] like Figure 1 As shown, this part corresponds to step S1, which includes the following:

[0061] 1. Preprocess the video file: Remove the intro and outro of the video file, extract two keyframes, and save them as the first keyframe image;

[0062] 2. Determine if the first keyframe image is a video with a top-bottom structure; if yes, proceed to step 3; if no, proceed to step 4.

[0063] Step 2 specifically involves the following steps: The first keyframe image has a height of h and a width of w. It is cut at 0.5h from the first keyframe image, resulting in two images, an upper image and a lower image, each with a height of 0.5h and a width of w. The ORB feature point matching + mean hash algorithm is used to calculate the similarity between the cut upper and lower images. If the similarity calculation result is greater than the upper / lower structure threshold, the two images are considered similar, and the video file is determined to have an upper / lower structure. The cut upper image is then saved, and the process proceeds to step 3. Otherwise, the process proceeds to step 4.

[0064] 3. Sequentially determine whether the first keyframe image is a 360-degree video or a 180-degree video. If it is a 360-degree video, return that the video file is a 360-degree video in 3D. If it is a 180-degree video, return that the video file is a 180-degree video in 3D. If neither is true, return that the video file is a normal video in 3D.

[0065] Step 3 specifically includes the following:

[0066] 3.1. Extract the leftmost 5 columns of pixels from the top image and save them as Image 1, and the rightmost 5 columns of pixels and save them as Image 2. Convert Image 1 and Image 2 into two 64-dimensional vectors. Calculate the similarity between these two vectors using a cosine similarity + mean hash algorithm. If the similarity is greater than the 360-degree video vertical 3D threshold, the two images are considered similar, and the video file is determined to be a 360-degree video vertical 3D. Otherwise, proceed to step 3.2.

[0067] 3.2 If the aspect ratio of the top image is equal to 1 or 2, the video file is determined to be a 180-degree vertical 3D video; otherwise, the video file is determined to be a regular vertical 3D video.

[0068] 4. Determine if the first keyframe image is a left-right structured video; if yes, proceed to step 5; if no, proceed to step 6; (If it is not a top-bottom structure, calculate whether it is a left-right structure. A left-right structured video is composed of two extremely similar images captured by two cameras, so calculating the left-right similarity can determine whether it is a left-right structure).

[0069] Step 4 specifically involves the following steps: The first keyframe image has a height of h and a width of w. It is then cut at 0.5w from the first keyframe image, resulting in a left image and a right image with a height of h and a width of 0.5w. The ORB feature point matching + mean hash algorithm is used to calculate the similarity between the cut left and right images. If the similarity calculation result is greater than the left-right structure threshold, it indicates that the two images are similar, and the video file is determined to be a left-right structure. The cut left image is then saved.

[0070] 5. Sequentially determine whether the first keyframe image is a 3D YTB video, a 360-degree video, or a 180-degree video. If it is a 3D YTB video, return that the video file is a 3D YTB video; if it is a 360-degree video, return that the video file is a 3D 360-degree video; if it is a 180-degree video, return that the video file is a 3D 180-degree video. If it is none of the three, return that the video file is a normal 3D video.

[0071] Step 5 specifically includes the following:

[0072] 5.1 Project the left image onto a six-sided cube. Take two columns of pixels at the junction of two adjacent faces of the cube, convert the two columns of pixels into two 64-dimensional vectors, and calculate the cosine similarity between the two vectors. If the cosine similarity is greater than the ytb video 3D threshold, then the video file is determined to be a ytb video 3D; otherwise, proceed to step 5.2 (Since ytb videos project 3D images onto a six-sided cube, the images at the junctions of the 12 edges of the cube, i.e., every two faces, are extremely similar. Therefore, take two columns of pixels at the junction of two adjacent faces, convert the two columns of pixels into two 64-dimensional vectors, and then calculate the cosine similarity between the two vectors. For example, if the similarity is greater than 0.65, then it is determined to be a ytb video 3D).

[0073] 5.2. Extract the leftmost 5 columns of pixels from the left image and save them as image 3, and the rightmost 5 columns of pixels and save them as image 4. Convert images 3 and 4 into two 64-dimensional vectors. Calculate the similarity between these two vectors using a cosine similarity + mean hash algorithm. If the similarity is greater than the 360-degree video horizontal 3D threshold, the two images are considered similar, and the video file is determined to be a 360-degree video horizontal 3D. Otherwise, proceed to step 5.3.

[0074] 5.3 If the aspect ratio of the left image is equal to 1 or 2, the video file is determined to be a 180-degree horizontal 3D video; otherwise, the video file is determined to be a normal horizontal 3D video.

[0075] 6. Sequentially determine whether the first keyframe image is a YTB video, a 360-degree video, or a 180-degree video. If it is a YTB video, return the video file as a YTB video; if it is a 360-degree video, return the video file as a 360-degree video; if it is a 180-degree video, return the video file as a 180-degree video; if it is none of the three, return the video file as a normal video.

[0076] Step 6 specifically includes the following steps:

[0077] 6.1 Project the first keyframe image into a six-sided cube. Take two columns of pixels at the junction of two adjacent faces of the six-sided cube. Convert the two columns of pixels into two 64-dimensional vectors. Calculate the cosine similarity between the two vectors. If the cosine similarity calculation result is greater than the ytb video threshold, then the video file is determined to be a ytb video; otherwise, proceed to step 6.2.

[0078] 6.2. Extract the leftmost 5 columns of pixels from the first keyframe image and save them as image 5, and the rightmost 5 columns of pixels and save them as image 6. Convert images 5 and 6 into two 64-dimensional vectors. Calculate the similarity between these two vectors using a cosine similarity + mean hash algorithm. If the similarity is greater than the 360-degree video threshold, the two images are considered similar, and the video file is determined to be a 360-degree video; otherwise, proceed to step 6.3.

[0079] 6.3 If the aspect ratio of the first keyframe image is equal to 1 or 2, the video file is determined to be a 180-degree video; otherwise, the video file is determined to be a normal video.

[0080] In this embodiment, the ORB feature point matching + mean hash algorithm is used to determine whether the video is a left-right or top-bottom structure. The ORB feature consists of key points and descriptors. Its key points are called "Oriented FAST", which is an improved FAST corner point. Its descriptor is BRIEF. The extraction of ORB features is divided into the following two steps: (1) FAST corner point extraction: find the corner points in the image. Compared with the original FAST algorithm, ORB calculates the main direction of the feature points, which adds rotation invariance to the BRIEF descriptor; (2) BRIEF descriptor: describe the pixel area around the key points found in the previous step. Since BRIEF is very sensitive to image rotation, ORB improves BRIEF by using the direction information calculated in the previous step to enhance the rotation invariance of BRIEF.

[0081] The ORB feature point matching + mean hashing algorithm works as follows: Key points are extracted from the two images obtained through segmentation. These key points are compared and matched to calculate the matching degree. Then, mean hashing is applied to both images to calculate the Hamming distance. If the calculated matching degree is greater than a matching threshold and the Hamming distance is less than a distance threshold, the two images are considered similar. For example, if the matching degree is greater than 0.2 and the Hamming distance is less than 8, the two images are considered similar.

[0082] In this embodiment, the cosine similarity + mean hash algorithm is suitable for calculating local image similarity, mainly used in judging 360 and YTB type videos. If the three channel values ​​of the image are directly used to calculate cosine similarity, the computational load is too large and the performance is poor, so the image-to-vector method needs improvement. The method for extracting vectors is as follows: Each of the three primary colors can take values ​​from 0 to 255. These values ​​are divided into four partitions: [0-63], [64-127], [128-191], and [191-255]. The partition of each primary color is calculated. For example, if a pixel's triplet is (64, 88, 133), it is converted to a partition corresponding to (1, 1, 3). Then, this partition value is mapped to an integer between 0 and 63. The calculation formula is: 1*42 +1*4 1 +3*4 0 If the value is 23, then the element with index 23 in the 64-dimensional vector corresponding to this pixel is used to calculate the indices of the left and right pixels. Then, the sum of the number of pixels at each index is calculated, resulting in the 64-dimensional vector corresponding to the image. Next, cosine similarity is calculated between the 64-dimensional vectors of the two images. Then, mean hashing is applied to the two images to calculate the Hamming distance. After obtaining the cosine similarity and Hamming distance, the two values ​​are combined for judgment.

[0083] The cosine similarity + mean hash algorithm works as follows: The range of values ​​for each of the three primary colors (0-255) is divided into four partitions: [0-63], [64-127], [128-191], and [191-255]. The partition for each primary color is calculated, and this partition value is mapped to an integer between 0 and 63. The indices of the left and right pixels are calculated, and the sum of the points corresponding to each index is obtained to acquire a 64-dimensional vector for the image. Cosine similarity is calculated for the 64-dimensional vectors of the two images obtained from the segmentation. Then, mean hashing is applied to the two images to calculate the Hamming distance. If the cosine similarity is greater than a cosine similarity threshold and the Hamming distance is less than a distance threshold, the two images are considered similar. For example, a cosine similarity greater than 0.65 and a Hamming distance less than 8 indicate that the two images are similar.

[0084] In this embodiment, various thresholds can be set according to actual needs in order to better meet those needs.

[0085] II. Model Recognition

[0086] like Figure 2 and Figure 3 As shown, this part corresponds to step S2, which specifically includes the following:

[0087] 1. Preprocess the video file: Remove the intro and outro of the video file, extract a keyframe, and save it as the second keyframe image;

[0088] 2. Preprocess the second keyframe image; Step 2 specifically includes the following:

[0089] 2.1 The transforms.Resize function crops the image of the second keyframe;

[0090] 2.2 Use the Cutout and Mixup functions to perform data augmentation on the cropped images to improve model performance;

[0091] 2.3. Use the ToTensor function to tensor the data-augmented image;

[0092] 2.4. Use the transforms.Normalize function to normalize the tensor-quantized images, calculate the average and standard values ​​of the overall image dataset, and accelerate the convergence of the model and improve the accuracy.

[0093] 3. Input the preprocessed second keyframe image into the Swin transformer model for image classification, and return the probability value of each category. The category with the highest probability value is the category to which the video file belongs. Step 3 specifically includes the following:

[0094] 3.1 The height of the second keyframe image is H and the width is W; the preprocessed image data is input into the PatchMerging layer. The patch partition divides the input image HxWx3 into a set of non-overlapping patches, where each patch is 4x4 in size, the feature dimension of each patch is 4x4x3=48, and the number of patch blocks is H / 4xW / 4.

[0095] 3.2. Input the image data output from the Patch Merging layer into the LineEmbedding and SwintransformerBlock layers. The linear embedding layer transforms the feature dimension of the divided patch into C, and then sends it into the Swin Transformer Block layer. Through a patch merging process, the input is merged into adjacent patches in a 2x2 pattern, making the number of patch blocks H / 8xW / 8 and the feature dimension 4C.

[0096] 3.3 The image data output from the LineEmbedding and SwintransformerBlock layers is fed into the PatchMerging and SwintransformerBlock layers. The PatchMerging and SwintransformerBlock layers perform fine-grained image feature extraction through a self-attention mechanism. This self-attention mechanism calculates self-attention based on windows, with each window containing 7*7 patches by default. The self-attention calculation is performed within these windows. The complexity is calculated using the following formula.

[0097]

[0098] Where h, ω, and C′ represent the height, width, and number of channels of the feature map, respectively; Patch Merging and SwintransformerBlock layers extract fine-grained image features through a self-attention mechanism, which has a very good processing effect on VR video format classification scenarios where the features to be classified are not obvious. At the same time, the complexity calculated by the above formula is much smaller than that of the original formula, reducing the inference time of the model and facilitating online and mobile deployment. It is very suitable for deploying the model on VR mobile devices.

[0099] S234. Input the image data output from the Patch Merging and SwintransformerBlock layers into the MLP layer, and calculate the probability value of each category using the Softmax function. The category with the highest probability value is the category to which the video file belongs.

[0100]

[0101] Among them, z i Let K be the output value of the i-th category after processing the image data through Patch Merging and the SwintransformerBlock layer, where K is the number of categories. The Softmax function is a normalization exponential function, a generalization of the binary classification function sigmoid to multi-class classification, aiming to represent the results of multi-class classification as probabilities. Using the Softmax function, the output values ​​of multi-class classification can be converted into a probability distribution with a sum of 1 within the range [0,1], achieving the effect of multi-class classification in VR video formats.

[0102] III. Comprehensive Judgment of Identification Results

[0103] This part corresponds to step S3. If the results of policy identification and model identification are consistent, the result of policy identification is returned; if the results of policy identification and model identification are inconsistent, the result of model identification shall prevail.

[0104] By adopting the above-disclosed technical solution of this invention, the following beneficial effects are obtained:

[0105] This invention provides a video type intelligent recognition method based on VR devices. By using computer vision and image correlation technology, the method identifies the type of video. After obtaining the video type, the VR video player can directly render the correct video.

[0106] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent video type recognition based on VR devices, characterized in that: Includes the following steps, S1. Perform policy recognition on the video file and obtain the policy recognition results; S1 includes, S11. Preprocess the video file: Remove the intro and outro of the video file, extract two keyframes, and save them as the first keyframe image. S12. Determine whether the first keyframe image is a video with a top-bottom structure; if so, proceed to step S13. If not, proceed to step S14; S13. Sequentially determine whether the first keyframe image is a 360-degree video or a 180-degree video. If it is a 360-degree video, return that the video file is a 360-degree video in 3D. If it is a 180-degree video, return that the video file is a 180-degree video in 3D. If neither is true, return that the video file is a normal video in 3D. S14. Determine whether the first keyframe image is a video with a left-right structure; if yes, proceed to step S15; if no, proceed to step S16. S15. Sequentially determine whether the first keyframe image is a 3D ytb video, a 360-degree video, or a 180-degree video. If it is a 3D ytb video, return that the video file is a 3D ytb video; if it is a 360-degree video, return that the video file is a 360-degree video (left and right 3D); if it is a 180-degree video, return that the video file is a 180-degree video (left and right 3D); if it is none of the three, return that the video file is a normal video (left and right 3D). S16. Sequentially determine whether the first keyframe image is a ytb video, a 360-degree video, or a 180-degree video. If it is a ytb video, return that the video file is a ytb video; if it is a 360-degree video, return that the video file is a 360-degree video; if it is a 180-degree video, return that the video file is a 180-degree video. If none of the three conditions are met, the video file will be returned as a normal video. S2. Perform model recognition on the video file and obtain the model recognition results; S2 includes, S21. Preprocess the video file: After removing the beginning and end of the video file, extract a key frame and save it as the second key frame image. S22. Preprocess the second keyframe image; S23. Input the preprocessed second keyframe image into the Swin transformer model for image classification and return the probability value of each category. The category with the highest probability value is the category to which the video file belongs. S3. Determine whether the strategy identification result and the model identification result are the same. If they are the same, the strategy identification result shall be taken as the final identification result. If they are different, the model recognition result will be used as the final recognition result.

2. The intelligent video type recognition method based on VR devices according to claim 1, characterized in that: Step S12 specifically involves cutting the first keyframe image, which has a height of h and a width of w, from 0.5h, into two images: an upper image and a lower image, each with a height of 0.5h and a width of w. The ORB feature point matching and mean hash algorithm is used to calculate the similarity between the cut upper and lower images. If the similarity result is greater than the upper / lower structure threshold, the two images are considered similar, and the video file is determined to have an upper / lower structure. The cut upper image is then saved, and the process proceeds to step S13. Otherwise, the process proceeds to step S14.

3. The intelligent video type recognition method based on VR devices according to claim 2, characterized in that: Step S13 specifically includes the following: S131. Extract the leftmost 5 columns of pixels from the top image and save them as Image 1, and the rightmost 5 columns of pixels and save them as Image 2. Convert Image 1 and Image 2 into two 64-dimensional vectors. Calculate the similarity between these two vectors using a cosine similarity + mean hash algorithm. If the similarity is greater than the 360-degree video vertical 3D threshold, the two images are considered similar, and the video file is determined to be a 360-degree video vertical 3D. Otherwise, proceed to step S132. S132. If the aspect ratio of the upper image is equal to 1 or 2, the video file is determined to be a 180-degree vertical 3D video; otherwise, the video file is determined to be a normal vertical 3D video.

4. The intelligent video type recognition method based on VR devices according to claim 3, characterized in that: Step S14 specifically involves the following steps: The first keyframe image has a height of h and a width of w. It is cut at 0.5w from the first keyframe image, resulting in a left image and a right image with a height of h and a width of 0.5w. The ORB feature point matching + mean hash algorithm is used to calculate the similarity between the cut left and right images. If the similarity calculation result is greater than the left-right structure threshold, it indicates that the two images are similar, and the video file is determined to be a left-right structure. The cut left image is then saved.

5. The intelligent video type recognition method based on VR devices according to claim 4, characterized in that: Step S15 specifically includes the following: S151. Project the left image onto a six-sided cube. Take two columns of pixels at the junction of two adjacent faces of the six-sided cube. Convert the two columns of pixels into two 64-dimensional vectors. Calculate the cosine similarity between the two vectors. If the cosine similarity calculation result is greater than the ytb video 3D threshold, then the video file is determined to be a ytb video 3D. Otherwise, proceed to step S152; S152. Extract the leftmost 5 columns of pixels from the left image and save them as image 3, and the rightmost 5 columns of pixels and save them as image 4. Convert image 3 and image 4 into two 64-dimensional vectors. Calculate the similarity between these two vectors using a cosine similarity + mean hash algorithm. If the similarity is greater than the 360-degree video left and right 3D threshold, the two images are considered similar, and the video file is determined to be a 360-degree video left and right 3D. Otherwise, proceed to step S153. S153. If the aspect ratio of the left image is equal to 1 or 2, the video file is determined to be a 180-degree horizontal 3D video; otherwise, the video file is determined to be a normal horizontal 3D video.

6. The intelligent video type recognition method based on VR devices according to claim 5, characterized in that: Step S16 specifically includes the following steps: S161. Project the first keyframe image into a six-sided cube. Take two columns of pixels at the junction of two adjacent faces of the six-sided cube. Convert the two columns of pixels into two 64-dimensional vectors. Calculate the cosine similarity between the two vectors. If the cosine similarity calculation result is greater than the ytb video threshold, then determine that the video file is a ytb video; otherwise, proceed to step S162. S162. Extract the leftmost 5 columns of pixels from the first keyframe image and save it as image 5, and the rightmost 5 columns of pixels and save it as image 6. Convert image 5 and image 6 into two 64-dimensional vectors. Calculate the similarity between the two vectors using the cosine similarity + mean hash algorithm. If the similarity is greater than the 360-degree video threshold, it means that the two images are similar, and the video file is determined to be a 360-degree video; otherwise, proceed to step S163. S163. If the aspect ratio of the first keyframe image is equal to 1 or 2, then the video file is determined to be a 180-degree video; otherwise, the video file is determined to be a normal video.

7. The intelligent video type recognition method based on VR devices according to any one of claims 2-6, characterized in that: The ORB feature point matching + mean hashing algorithm is as follows: Key points are extracted from the two images obtained from the segmentation. The key points extracted from the two images are compared and matched to calculate the matching degree. The two images are then subjected to mean hashing to calculate the Hamming distance. If the calculated matching degree is greater than the matching threshold and the Hamming distance is less than the distance threshold, it means that the two images are similar.

8. The intelligent video type recognition method based on VR devices according to any one of claims 3-6, characterized in that: The cosine similarity + mean hash algorithm is as follows: The possible values ​​of each of the three primary colors, ranging from 0 to 255, are divided into four partitions: [0–63], [64–127], [128–191], and [191–255]. The partition corresponding to each primary color is calculated, and this partition value is mapped to an integer between 0 and 63. The indices of the left and right pixels are calculated, and the number of pixels corresponding to each index is summed to obtain a 64-dimensional vector for the image. Cosine similarity is calculated for the 64-dimensional vectors of the two images obtained from the segmentation. Then, mean hashing is performed on the two images to calculate the Hamming distance. If the cosine similarity is greater than the cosine similarity threshold and the Hamming distance is less than the distance threshold, the two images are considered similar.

9. The intelligent video type recognition method based on VR devices according to claim 1, characterized in that: Step S22 specifically includes the following: S221, the transforms.Resize function crops the image of the second keyframe; S222. Use the Cutout and Mixup functions to perform data augmentation on the cropped image; S223. Use the ToTensor function to tensorize the data-augmented image; S224. Use the transforms.Normalize function to normalize the tensor image.

10. The intelligent video type recognition method based on VR devices according to claim 9, characterized in that: Step S23 specifically includes the following: S231. The height of the second keyframe image is H and the width is W. The preprocessed image data is input into the PatchMerging layer. The patch partition divides the input image HxWx3 into a set of non-overlapping patches, where each patch is 4x4 in size, the feature dimension of each patch is 4x4x3=48, and the number of patch blocks is H / 4xW / 4. S232. Input the image data output from the Patch Merging layer into the LineEmbedding and SwintransformerBlock layers. The linear embedding layer transforms the feature dimension of the divided patch into C, and then sends it into the Swin Transformer Block layer. Through a patch merging, the input is merged into adjacent patches in a 2x2 pattern, making the number of patch blocks H / 8xW / 8 and the feature dimension 4C. S233. The image data output from the LineEmbedding and SwintransformerBlock layers is fed into the PatchMerging and SwintransformerBlock layers. The PatchMerging and SwintransformerBlock layers perform fine-grained image feature extraction through a self-attention mechanism. The self-attention mechanism calculates self-attention based on windows, with each window having a default of 7*7 patches. The self-attention calculation is completed within these windows. During the self-attention calculation process, the complexity is... The calculation formula is as follows: ， in, , and These represent the height, width, and number of channels of the feature map, respectively. S234. Input the image data output from the Patch Merging and SwintransformerBlock layers into the MLP layer, and calculate the probability value of each category using the Softmax function. The category with the highest probability value is the category to which the video file belongs. The formula for the Softmax function is as follows: ， in, is the output value of the i-th category after the image data is processed by Patch Merging and SwintransformerBlock layers, and K is the number of categories.

Images