VR body sensing detection optimization method and system

Abstract

The invention discloses a VR somatosensory detection optimization method and system, which belongs to the field of image optimization technology. The invention determines whether the predicted frame grayscale image is a similar image corresponding to the current frame grayscale image through the information change value and the feature point difference value between the current frame grayscale image and the predicted frame grayscale image. If not, the predicted frame grayscale image is further determined in combination with the information change value and the feature point difference value to determine the type of change relative to the current frame grayscale image. If it is a global change, the predicted gaze area is analyzed according to the predicted gaze reliability value of each segmented area in the predicted frame grayscale image. If it is an obvious local change, the predicted gaze area is analyzed according to the number of non-matching feature points in each segmented area. The predicted gaze area is analyzed with different parameters according to different types of changes, the prediction accuracy of the predicted gaze area is improved, and the predicted gaze area is rendered in high definition to optimize the VR visual somatosensory.

Description

Technical Field

[0001] The present invention relates to the field of image optimization technology, and in particular to a VR somatosensory detection optimization method and system. Background Art

[0002] With the development of science and technology and social economy, VR technology has been used in various industries, such as VR videos. In order to give users a good visual experience, each frame of the VR video is rendered in high definition. However, if all areas in each image are rendered in high definition, the amount of calculation is huge. Therefore, the existing technology relies on predicting the user's gaze area in each frame of the image and rendering these areas in high definition.

[0003] How to accurately predict the user's gaze area in each frame of the image is an urgent problem that needs to be solved. Summary of the invention

[0004] The present invention aims to solve the technical problems existing in the prior art. The present invention provides a VR somatosensory detection optimization method, comprising the following steps:

[0005] Obtain the grayscale image of each frame of the VR video, extract the feature point information of each grayscale image, and perform image segmentation on each grayscale image to obtain the segmented area of ​​each frame of the grayscale image;

[0006] The previous grayscale image in any two adjacent grayscale images is used as the current grayscale image, and the next grayscale image is used as the predicted grayscale image;

[0007] Analyze the information change value △E of the predicted frame grayscale image relative to the corresponding current frame grayscale image and the feature point difference value TC between the predicted frame grayscale image and the corresponding current frame grayscale image, and judge whether the predicted frame grayscale image is a similar image of the corresponding current frame grayscale image according to the information change value and the feature point difference value. If so, judge that the predicted fixation area of ​​the predicted frame grayscale image is the same as the fixation area of ​​the corresponding current frame grayscale image. If not, judge the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image according to the information change value and the feature point difference value, and the change type includes obvious local change and global change;

[0008] If the change type is a global change, the predicted gaze reliability value L of each segmented area in the predicted frame grayscale image is calculated, and the segmented area with the largest predicted gaze reliability value is used as the predicted gaze area of ​​the predicted frame grayscale image. If the change type is an obvious local change, the number of non-matching feature points in each segmented area in the predicted frame grayscale image relative to the corresponding current frame grayscale image is analyzed according to the position of the feature points in the current frame grayscale image and the position in the predicted frame grayscale image, and the segmented area with the largest number of non-matching feature points in the predicted frame grayscale image is used as the predicted gaze area of ​​the predicted frame grayscale image.

[0009] High-definition rendering of predicted fixation regions.

[0010] Furthermore, the specific calculation formula of the information change value ΔE is:

[0011] ,

[0012] ,

[0013] ;

[0014] Among them, E1 represents the image information entropy of the corresponding current frame grayscale image, E2 represents the image information entropy of the corresponding predicted frame grayscale image, u11 represents the maximum grayscale value contained in the corresponding current frame grayscale image, and u12 represents the minimum grayscale value contained in the corresponding current frame grayscale image. It is represented by the probability of the pixel with gray value i appearing in the corresponding current frame gray image, u21 is represented by the maximum gray value contained in the corresponding predicted frame gray image, and u22 is represented by the minimum gray value contained in the corresponding predicted frame gray image. It is expressed as the probability of the pixel with gray value i appearing in the corresponding predicted frame gray image.

[0015] Furthermore, the specific calculation formula of the characteristic point difference value TC is:

[0016] ,

[0017] ;

[0018] Among them, k1 represents the first preset weight, k2 represents the second preset weight, TS1 represents the number of feature points of the corresponding current frame grayscale image, TS2 represents the number of feature points of the corresponding predicted frame grayscale image, and FTS represents the number of non-matching feature points in the predicted frame grayscale image relative to the corresponding current frame grayscale image.

[0019] Furthermore, judging whether the predicted frame grayscale image is a similar image to the corresponding current frame grayscale image according to the information change value is specifically as follows:

[0020] If the information change value is less than the preset change threshold and the feature point difference value is less than the preset difference threshold, the corresponding predicted frame grayscale image belongs to a similar image of the corresponding current frame grayscale image.

[0021] Furthermore, judging the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image according to the information change value and the feature point difference value is specifically:

[0022] If the information change value is greater than or equal to the preset change threshold and the feature point difference value is less than the preset difference threshold, or the information change value is less than the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, then the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is an obvious local change;

[0023] If the information change value is greater than or equal to the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is a global change.

[0024] Furthermore, the predicted gaze reliability value is specifically calculated as follows:

[0025] ;

[0026] Wherein, M represents the number of adjacent segmented regions corresponding to the segmented region in the predicted frame grayscale image, and N represents the number of pixels in the corresponding segmented region in the predicted frame grayscale image. It represents the number of pixels in the mth adjacent segmentation area of ​​the corresponding segmentation area in the predicted frame grayscale image, e is a natural constant, Represents the mean grayscale value of the corresponding segmented area in the predicted frame grayscale image, It represents the mean gray value of the mth adjacent segmentation area corresponding to the segmentation area in the gray image of the predicted frame. Represents the information entropy of the corresponding segmented area in the predicted frame grayscale image, Represents the information entropy of the mth adjacent segmented area corresponding to the segmented area in the predicted frame grayscale image.

[0027] Furthermore, the method for obtaining the grayscale image of each frame of the VR video specifically includes the following steps:

[0028] Decomposing the VR video frame by frame to obtain an image of each frame of the VR video;

[0029] Grayscale processing is performed on each frame image of the VR video to obtain a corresponding grayscale image.

[0030] The present invention also provides a VR somatosensory detection optimization system, comprising:

[0031] An image acquisition module is used to acquire grayscale images of each frame of the VR video, and use the previous grayscale image of any two adjacent grayscale images as the current grayscale image, and the next grayscale image as the predicted grayscale image;

[0032] A feature point extraction module is used to extract feature point information of the grayscale image of each frame of the VR video;

[0033] A region segmentation module is used to segment each grayscale image and obtain the segmented region of each frame of grayscale image;

[0034] The first analysis module is used to analyze the information change value △E of the predicted frame grayscale image relative to the corresponding current frame grayscale image and the feature point difference value TC between the predicted frame grayscale image and the corresponding current frame grayscale image;

[0035] A judgment module is used to judge whether the predicted frame grayscale image is a similar image of the corresponding current frame grayscale image according to the information change value and the feature point difference value. If so, it is determined that the predicted fixation area of ​​the predicted frame grayscale image is the same as the fixation area of ​​the corresponding current frame grayscale image. If not, the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is judged according to the information change value and the feature point difference value, and the change type includes obvious local change and global change;

[0036] A second analysis module is used to determine the predicted fixation area of ​​the predicted frame grayscale image according to the type of change of the predicted frame grayscale image relative to the corresponding current frame grayscale image;

[0037] The rendering module is used to perform high-definition rendering on the predicted fixation area of ​​the predicted frame grayscale image.

[0038] Furthermore, judging whether the predicted frame grayscale image belongs to a similar image of the corresponding current frame grayscale image according to the information change value and the feature point difference value is specifically as follows:

[0039] If the information change value is less than the preset change threshold and the feature point difference value is less than the preset difference threshold, the corresponding predicted frame grayscale image belongs to a similar image of the corresponding current frame grayscale image;

[0040] The method of judging the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image according to the information change value and the feature point difference value is specifically:

[0041] If the information change value is greater than or equal to the preset change threshold and the feature point difference value is less than the preset difference threshold, or the information change value is less than the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, then the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is an obvious local change;

[0042] If the information change value is greater than or equal to the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is a global change.

[0043] Furthermore, judging the predicted fixation area of ​​the predicted frame grayscale image according to the type of change of the predicted frame grayscale image relative to the corresponding current frame grayscale image is specifically:

[0044] If the change type is a global change, the predicted gaze reliability value L of each segmented area in the predicted frame grayscale image is calculated, and the segmented area with the largest predicted gaze reliability value is used as the predicted gaze area of ​​the predicted frame grayscale image;

[0045] If the type of change is an obvious local change, the number of non-matching feature points in each segmented area in the predicted frame grayscale image relative to the corresponding current frame grayscale image is analyzed based on the position of the feature points in the current frame grayscale image and the position in the predicted frame grayscale image, and the segmented area with the largest number of non-matching feature points in the predicted frame grayscale image is used as the predicted attention area of ​​the predicted frame grayscale image.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention first determines whether the predicted frame grayscale image belongs to the similar image of the corresponding current frame grayscale image according to the information change value and the feature point difference value. If so, the attention area of ​​the current frame grayscale image is directly used as the predicted attention area of ​​the predicted frame grayscale image without further calculation and prediction, thereby ensuring the accuracy of the predicted attention area of ​​the predicted frame grayscale image as a similar image. If not, the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is determined according to the information change value and the feature point difference value. The information change value and the feature point difference value are combined for judgment to avoid the situation where the information change value mainly reflects the distribution of pixel values ​​but cannot The changes in local areas of the image are captured, which may lead to inaccurate judgment of the change types of the predicted frame grayscale image, thereby improving the prediction accuracy. At the same time, the types of changes are divided into obvious local changes and global changes. The predicted gaze reliability value calculated by the information entropy of each segmented area in the predicted frame grayscale image is used to predict the gaze area of ​​the predicted frame grayscale image belonging to the global change. The segmented area with the largest number of non-matching feature points in the predicted frame grayscale image is used as the predicted gaze area of ​​the predicted frame grayscale image. Different parameters are selected for judgment and prediction according to the inertial gaze habits corresponding to different types of changes, thereby further improving the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1It is a flow chart of a VR somatosensory detection optimization method of the present invention;

[0051] Figure 2 It is a structural block diagram of a VR somatosensory detection optimization system of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0054] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0055] Embodiment 1

[0056] See also Figure 1 As shown, the present invention provides a VR somatosensory detection optimization method, comprising:

[0057] S1. Obtain a grayscale image of each frame of a VR video, extract feature point information of each grayscale image, and perform image segmentation on each grayscale image to obtain a segmented area of ​​each frame of the grayscale image.

[0058] S2. The previous grayscale image frame of any two adjacent grayscale image frames is used as the current grayscale image frame, and the next grayscale image frame is used as the predicted grayscale image frame.

[0059] S3. Analyze the information change value △E of the predicted frame grayscale image relative to the corresponding current frame grayscale image and the feature point difference value TC between the predicted frame grayscale image and the corresponding current frame grayscale image, and judge whether the predicted frame grayscale image is a similar image of the corresponding current frame grayscale image according to the information change value and the feature point difference value. If so, determine that the predicted gaze area of ​​the predicted frame grayscale image is the same as the gaze area of ​​the corresponding current frame grayscale image. If not, judge the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image according to the information change value and the feature point difference value, and the change type includes obvious local changes and global changes.

[0060] S4, if the change type is a global change, then calculating the predicted gaze reliability value L of each segmented area in the predicted frame grayscale image, and taking the segmented area with the largest predicted gaze reliability value as the predicted gaze area of ​​the predicted frame grayscale image;

[0061] If the type of change is an obvious local change, the number of non-matching feature points in each segmented area in the predicted frame grayscale image relative to the corresponding current frame grayscale image is analyzed based on the position of the feature points in the current frame grayscale image and the position in the predicted frame grayscale image, and the segmented area with the largest number of non-matching feature points in the predicted frame grayscale image is used as the predicted attention area of ​​the predicted frame grayscale image.

[0062] Global changes are equivalent to video transition effects. There are large differences in image information entropy and feature points between the two frames. Therefore, when predicting the gaze area of ​​the next frame, it is impossible to associate it with the previous frame. Human gaze habits usually give priority to focusing on areas in the image that are significantly different from other areas. Therefore, in the case of global changes, this scheme makes judgments based on the grayscale values ​​and information entropy of each segmented area in the predicted frame grayscale image to obtain the predicted gaze reliability values.

[0063] S5. Perform high-definition rendering on the predicted gaze area.

[0064] For step S1, the grayscale image of each frame of the VR video is obtained, the feature point information of each grayscale image is extracted, and each grayscale image is segmented to obtain the segmented area of ​​each frame of the grayscale image:

[0065] The method for obtaining the grayscale image of each frame of the VR video specifically includes the following steps:

[0066] Decomposing the VR video frame by frame to obtain an image of each frame of the VR video;

[0067] Each frame image of the VR video is preprocessed, and each frame image is grayscaled to obtain a corresponding grayscale image.

[0068] The VR video is frame-decomposed to obtain the image of each frame of the VR video. The VR video file can be converted into an operable frame sequence through a video decoder. Common decoding libraries include libavcodec or GStreamer in FFmpeg, and then the image data is read frame by frame from the frame sequence. This can be achieved by setting the decoder to stop at each key frame or specified interval and extract the image data of the frame.

[0069] The feature point information of each grayscale image can be extracted by any feature point detection algorithm such as SIFT, SURF, Harris corner detection, etc. The specific implementation process of extracting feature point information by the above feature point detection algorithm belongs to the prior art and will not be repeated here. The feature point information includes the number of feature points and their locations, etc.

[0070] The image segmentation of the grayscale image of each frame of the VR video is performed, specifically, the image segmentation of the grayscale image of each frame of the VR video is performed by a region growing algorithm. This specific implementation process is a prior art and will not be repeated here.

[0071] In step S3, the specific calculation formula of the information change value △E is:

[0072] ,

[0073] ,

[0074] ;

[0075] Among them, E1 represents the image information entropy of the corresponding current frame grayscale image, E2 represents the image information entropy of the corresponding predicted frame grayscale image, u11 represents the maximum grayscale value contained in the corresponding current frame grayscale image, and u12 represents the minimum grayscale value contained in the corresponding current frame grayscale image. It is represented by the probability of the pixel with gray value i appearing in the corresponding current frame gray image, u21 is represented by the maximum gray value contained in the corresponding predicted frame gray image, and u22 is represented by the minimum gray value contained in the corresponding predicted frame gray image. It is expressed as the probability of the pixel with gray value i appearing in the corresponding predicted frame gray image.

[0076] The specific calculation formula of the characteristic point difference value TC is:

[0077] ,

[0078] ;

[0079] Wherein, k1 represents the first preset weight, k2 represents the second preset weight, TS1 represents the number of feature points of the corresponding current frame grayscale image, TS2 represents the number of feature points of the corresponding predicted frame grayscale image, FTS represents the number of non-matching feature points in the predicted frame grayscale image relative to the corresponding current frame grayscale image, and the feature points whose Euclidean distances with all feature points in the predicted frame grayscale image and the corresponding current frame grayscale image are greater than the preset distance threshold are the non-matching feature points in the predicted frame grayscale image relative to the corresponding current frame grayscale image;

[0080] For example, a feature point A in a predicted frame grayscale image has a position coordinate of (x1, y1), and a feature point B in the corresponding current frame grayscale image (x2, y2). The Euclidean distance between A and B is If the Euclidean distances between feature point A in the predicted frame grayscale image and all feature points in the corresponding current frame grayscale image are greater than a preset distance threshold, then feature point A is a non-matching feature point in the predicted frame grayscale image relative to the corresponding current frame grayscale image.

[0081] The step of judging whether the predicted frame grayscale image is a similar image to the corresponding current frame grayscale image according to the information change value and the feature point difference value is as follows:

[0082] If the information change value is less than the preset change threshold and the feature point difference value is less than the preset difference threshold, the corresponding predicted frame grayscale image belongs to a similar image of the corresponding current frame grayscale image.

[0083] The method of judging the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image according to the information change value and the feature point difference value is specifically:

[0084] If the information change value is greater than or equal to the preset change threshold and the feature point difference value is less than the preset difference threshold, or the information change value is less than the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, then the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is an obvious local change;

[0085] If the information change value is greater than or equal to the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is a global change.

[0086] The information change value reflects the change in image information entropy between two frames of images. When the image information entropy of two frames of images changes slightly, it usually means that the contents of the two images are very similar. However, the information entropy mainly reflects the distribution of pixel values ​​rather than the specific changes in pixel values. The information entropy cannot capture changes in local areas of the image. Even if the global information entropy of the image does not change much, the local area may have changed significantly. Therefore, this scheme combines the feature point difference value between the two images. The feature point is a specific position in the image that is unique and repeatable. Therefore, the combination of the information change value and the feature point difference value between the two images can reflect whether the local position in the image has changed significantly. When the information change value is large, it indicates that the image.

[0087] The specific calculation formula of the predicted gaze reliability value is:

[0088] ;

[0089] Wherein, M represents the number of adjacent segmented regions corresponding to the segmented region in the predicted frame grayscale image, and N represents the number of pixels in the corresponding segmented region in the predicted frame grayscale image. It represents the number of pixels in the mth adjacent segmentation area of ​​the corresponding segmentation area in the predicted frame grayscale image, e is a natural constant, Represents the mean grayscale value of the corresponding segmented area in the predicted frame grayscale image, It represents the mean gray value of the mth adjacent segmentation area corresponding to the segmentation area in the gray image of the predicted frame. Represents the information entropy of the corresponding segmented area in the predicted frame grayscale image, Represents the information entropy of the mth adjacent segmented area corresponding to the segmented area in the predicted frame grayscale image.

[0090] Generally speaking, the area with a larger mean gray value and a larger average information entropy of each pixel in the image is more likely to attract attention. As the initial attention value of the corresponding segmented area relative to the corresponding m-th adjacent segmented area, generally speaking, the larger the initial attention value, the more attention the corresponding segmented area will attract compared to the corresponding m-th adjacent segmented area. However, the effective reference degree of the initial attention value is affected by the size of the corresponding segmented area and its m-th adjacent segmented area. The closer the sizes of the two are, the higher the effective reference degree of the initial attention value is. Because the closer the number of pixels between the two segmented areas is, the error caused by the inconsistency of the number of pixels can be avoided when comparing the gray value mean and information entropy. Therefore, the formula introduces As an adjustment coefficient, it improves the effective reference of the final attention value, thereby improving the effectiveness of predicting the reliable value of fixation.

[0091] Embodiment 2

[0092] See also Figure 2 As shown, the present invention also provides a VR somatosensory detection optimization system, comprising:

[0093] An image acquisition module, a feature point extraction module, a region segmentation module, a first analysis module, a judgment module, a second analysis module and a rendering module, and a central processing unit that is in communication with the above modules.

[0094] An image acquisition module is used to acquire grayscale images of each frame of the VR video, and use the previous grayscale image of any two adjacent grayscale images as the current grayscale image, and the next grayscale image as the predicted grayscale image;

[0095] A feature point extraction module is used to extract feature point information of the grayscale image of each frame of the VR video;

[0096] A region segmentation module is used to segment each grayscale image and obtain the segmented region of each frame of grayscale image;

[0097] The first analysis module is used to analyze the information change value △E of the predicted frame grayscale image relative to the corresponding current frame grayscale image and the feature point difference value TC between the predicted frame grayscale image and the corresponding current frame grayscale image;

[0098] A judgment module is used to judge whether the predicted frame grayscale image is a similar image of the corresponding current frame grayscale image according to the information change value and the feature point difference value. If so, it is determined that the predicted fixation area of ​​the predicted frame grayscale image is the same as the fixation area of ​​the corresponding current frame grayscale image. If not, the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is judged according to the information change value and the feature point difference value, and the change type includes obvious local change and global change;

[0099] A second analysis module is used to determine the predicted fixation area of ​​the predicted frame grayscale image according to the type of change of the predicted frame grayscale image relative to the corresponding current frame grayscale image;

[0100] The rendering module is used to perform high-definition rendering on the predicted fixation area of ​​the predicted frame grayscale image.

[0101] The specific calculation formula of information change value △E is:

[0102] ,

[0103] ,

[0104] ;

[0105] Among them, E1 represents the image information entropy of the corresponding current frame grayscale image, E2 represents the image information entropy of the corresponding predicted frame grayscale image, u11 represents the maximum grayscale value contained in the corresponding current frame grayscale image, and u12 represents the minimum grayscale value contained in the corresponding current frame grayscale image. It is represented by the probability of the pixel with gray value i appearing in the corresponding current frame gray image, u21 is represented by the maximum gray value contained in the corresponding predicted frame gray image, and u22 is represented by the minimum gray value contained in the corresponding predicted frame gray image. It is expressed as the probability of the pixel with gray value i appearing in the corresponding predicted frame gray image.

[0106] The specific calculation formula of the characteristic point difference value TC is:

[0107] ,

[0108] ;

[0109] Among them, k1 represents the first preset weight, k2 represents the second preset weight, TS1 represents the number of feature points of the corresponding current frame grayscale image, TS2 represents the number of feature points of the corresponding predicted frame grayscale image, FTS represents the number of non-matching feature points in the predicted frame grayscale image relative to the corresponding current frame grayscale image, and the feature points whose Euclidean distances between the predicted frame grayscale image and all feature points in the corresponding current frame grayscale image are greater than the preset distance threshold are the non-matching feature points in the predicted frame grayscale image relative to the corresponding current frame grayscale image.

[0110] The step of judging whether the predicted frame grayscale image is a similar image to the corresponding current frame grayscale image according to the information change value is specifically as follows:

[0111] If the information change value is less than the preset change threshold and the feature point difference value is less than the preset difference threshold, the corresponding predicted frame grayscale image belongs to a similar image of the corresponding current frame grayscale image.

[0112] The method of judging the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image according to the information change value and the feature point difference value is specifically:

[0113] If the information change value is greater than or equal to the preset change threshold and the feature point difference value is less than the preset difference threshold, or the information change value is less than the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, then the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is an obvious local change;

[0114] If the information change value is greater than or equal to the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is a global change.

[0115] The step of determining the predicted fixation area of ​​the predicted frame grayscale image according to the type of change of the predicted frame grayscale image relative to the corresponding current frame grayscale image is specifically as follows:

[0116] If the change type is a global change, the predicted gaze reliability value L of each segmented area in the predicted frame grayscale image is calculated, and the segmented area with the largest predicted gaze reliability value is used as the predicted gaze area of ​​the predicted frame grayscale image;

[0117] If the type of change is an obvious local change, the number of non-matching feature points in each segmented area in the predicted frame grayscale image relative to the corresponding current frame grayscale image is analyzed based on the position of the feature points in the current frame grayscale image and the position in the predicted frame grayscale image, and the segmented area with the largest number of non-matching feature points in the predicted frame grayscale image is used as the predicted attention area of ​​the predicted frame grayscale image.

[0118] The specific calculation formula of the predicted gaze reliability value is:

[0119] ;

[0120] Wherein, M represents the number of adjacent segmented regions corresponding to the segmented region in the predicted frame grayscale image, and N represents the number of pixels in the corresponding segmented region in the predicted frame grayscale image. It represents the number of pixels in the mth adjacent segmentation area of ​​the corresponding segmentation area in the predicted frame grayscale image, e is a natural constant, Represents the mean grayscale value of the corresponding segmented area in the predicted frame grayscale image, It represents the mean gray value of the mth adjacent segmentation area corresponding to the segmentation area in the gray image of the predicted frame. Represents the information entropy of the corresponding segmented area in the predicted frame grayscale image, Represents the information entropy of the mth adjacent segmented area corresponding to the segmented area in the predicted frame grayscale image.

[0121] Embodiment 3

[0122] The present invention also provides an electronic device, including: a processor, a sending device, an input device, an output device and a memory. The processor can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the technical solution provided in the embodiment of the present application. The memory can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM), and is used to store computer program code. The computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes a method as described in any of the above possible implementation methods.

[0123] Embodiment 4

[0124] The present invention also provides a computer-readable storage medium, in which a computer program is stored. The computer program includes program instructions. When the program instructions are executed by a processor of an electronic device, the processor executes a method as described in any possible implementation manner.

[0125] The beneficial effects of the present invention are:

[0126] The present invention first determines whether the predicted frame grayscale image belongs to the similar image of the corresponding current frame grayscale image according to the information change value and the feature point difference value. If so, the attention area of ​​the current frame grayscale image is directly used as the predicted attention area of ​​the predicted frame grayscale image without further calculation and prediction, thereby ensuring the accuracy of the predicted attention area of ​​the predicted frame grayscale image as a similar image. If not, the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is determined according to the information change value and the feature point difference value. The information change value and the feature point difference value are combined for judgment to avoid the situation where the information change value mainly reflects the distribution of pixel values ​​without The method captures the changes in local areas of the image, which leads to inaccurate judgment of the change types of the predicted frame grayscale image, improves the prediction accuracy, and at the same time divides the change types into obvious local changes and global changes. The predicted gaze reliability value calculated by the information entropy of each segmented area in the predicted frame grayscale image is used to predict the gaze area of ​​the predicted frame grayscale image belonging to the global change. The segmented area with the largest number of non-matching feature points in the predicted frame grayscale image is used as the predicted gaze area of ​​the predicted frame grayscale image. According to the gaze habits corresponding to different types of changes, different parameters are selected for judgment and prediction, so as to further improve the prediction accuracy.

[0127] In the description of the specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0128] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (RandomAccess Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0129] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A VR somatosensory detection optimization method, characterized in that: The following steps are involved: Obtain the grayscale image of each frame of the VR video, extract the feature point information of each grayscale image, and perform image segmentation on each grayscale image to obtain the segmented area of ​​each frame of the grayscale image; The previous grayscale image in any two adjacent grayscale images is used as the current grayscale image, and the next grayscale image is used as the predicted grayscale image; Analyze the information change value △E of the predicted frame grayscale image relative to the corresponding current frame grayscale image and the feature point difference value TC between the predicted frame grayscale image and the corresponding current frame grayscale image, and judge whether the predicted frame grayscale image is a similar image of the corresponding current frame grayscale image according to the information change value and the feature point difference value. If so, judge that the predicted fixation area of ​​the predicted frame grayscale image is the same as the fixation area of ​​the corresponding current frame grayscale image. If not, judge the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image according to the information change value and the feature point difference value, and the change type includes obvious local change and global change; If the change type is a global change, the predicted gaze reliability value L of each segmented area in the predicted frame grayscale image is calculated, and the segmented area with the largest predicted gaze reliability value is used as the predicted gaze area of ​​the predicted frame grayscale image. If the change type is an obvious local change, the number of non-matching feature points in each segmented area in the predicted frame grayscale image relative to the corresponding current frame grayscale image is analyzed according to the position of the feature points in the current frame grayscale image and the position in the predicted frame grayscale image, and the segmented area with the largest number of non-matching feature points in the predicted frame grayscale image is used as the predicted gaze area of ​​the predicted frame grayscale image. High-definition rendering of the predicted fixation area; The specific formula of the information change value △E is: ， ， ， Among them, E1 represents the image information entropy of the corresponding current frame grayscale image, E2 represents the image information entropy of the corresponding predicted frame grayscale image, u11 represents the maximum grayscale value contained in the corresponding current frame grayscale image, and u12 represents the minimum grayscale value contained in the corresponding current frame grayscale image. It is represented by the probability of the pixel with gray value i appearing in the corresponding current frame gray image, u21 is represented by the maximum gray value contained in the corresponding predicted frame gray image, and u22 is represented by the minimum gray value contained in the corresponding predicted frame gray image. It is expressed as the probability of the pixel with gray value i appearing in the corresponding predicted frame gray image; The specific formula of the characteristic point difference value TC is: ， ， Wherein, k1 represents the first preset weight, k2 represents the second preset weight, TS1 represents the number of feature points of the corresponding current frame grayscale image, TS2 represents the number of feature points of the corresponding predicted frame grayscale image, and FTS represents the number of non-matching feature points in the predicted frame grayscale image relative to the corresponding current frame grayscale image; The specific formula for predicting the reliable value of gaze is: ， Wherein, M represents the number of adjacent segmented regions corresponding to the segmented region in the predicted frame grayscale image, and N represents the number of pixels in the corresponding segmented region in the predicted frame grayscale image. It represents the number of pixels in the mth adjacent segmentation area of ​​the corresponding segmentation area in the predicted frame grayscale image, e is a natural constant, Represents the mean grayscale value of the corresponding segmented area in the predicted frame grayscale image, It represents the mean gray value of the mth adjacent segmentation area corresponding to the segmentation area in the gray image of the predicted frame. Represents the information entropy of the corresponding segmented area in the predicted frame grayscale image, Represents the information entropy of the mth adjacent segmented area corresponding to the segmented area in the predicted frame grayscale image.

2. The VR somatosensory detection optimization method according to claim 1, characterized in that: The step of judging whether the predicted frame grayscale image is a similar image to the corresponding current frame grayscale image according to the information change value is specifically as follows: If the information change value is less than the preset change threshold and the feature point difference value is less than the preset difference threshold, the corresponding predicted frame grayscale image belongs to a similar image of the corresponding current frame grayscale image.

3. The VR somatosensory detection optimization method according to claim 1, characterized in that: The method of judging the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image according to the information change value and the feature point difference value is specifically: If the information change value is greater than or equal to the preset change threshold and the feature point difference value is less than the preset difference threshold, or the information change value is less than the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, then the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is an obvious local change; If the information change value is greater than or equal to the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is a global change.

4. The VR somatosensory detection optimization method according to claim 1, characterized in that: The method for obtaining the grayscale image of each frame of the VR video specifically includes the following steps: Decomposing the VR video frame by frame to obtain an image of each frame of the VR video; Grayscale processing is performed on each frame image of the VR video to obtain a corresponding grayscale image.

5. A VR somatosensory detection optimization system, characterized in that: include: An image acquisition module is used to acquire grayscale images of each frame of the VR video, and use the previous grayscale image of any two adjacent grayscale images as the current grayscale image, and the next grayscale image as the predicted grayscale image; A feature point extraction module is used to extract feature point information of the grayscale image of each frame of the VR video; A region segmentation module is used to segment each grayscale image and obtain the segmented region of each frame of grayscale image; The first analysis module is used to analyze the information change value △E of the predicted frame grayscale image relative to the corresponding current frame grayscale image and the feature point difference value TC between the predicted frame grayscale image and the corresponding current frame grayscale image; A judgment module is used to judge whether the predicted frame grayscale image is a similar image of the corresponding current frame grayscale image according to the information change value and the feature point difference value. If so, it is determined that the predicted fixation area of ​​the predicted frame grayscale image is the same as the fixation area of ​​the corresponding current frame grayscale image. If not, the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is judged according to the information change value and the feature point difference value, and the change type includes obvious local change and global change; A second analysis module is used to determine the predicted fixation area of ​​the predicted frame grayscale image according to the type of change of the predicted frame grayscale image relative to the corresponding current frame grayscale image; If the change type is a global change, the predicted gaze reliability value L of each segmented area in the predicted frame grayscale image is calculated, and the segmented area with the largest predicted gaze reliability value is used as the predicted gaze area of ​​the predicted frame grayscale image; if the change type is an obvious local change, the number of non-matching feature points in each segmented area in the predicted frame grayscale image relative to the corresponding current frame grayscale image is analyzed according to the position of the feature points in the current frame grayscale image and the position in the predicted frame grayscale image, and the segmented area with the largest number of non-matching feature points in the predicted frame grayscale image is used as the predicted gaze area of ​​the predicted frame grayscale image; A rendering module, used for performing high-definition rendering of the predicted fixation area of ​​the predicted frame grayscale image; The specific formula of the information change value △E is: ， ， ， Among them, E1 represents the image information entropy of the corresponding current frame grayscale image, E2 represents the image information entropy of the corresponding predicted frame grayscale image, u11 represents the maximum grayscale value contained in the corresponding current frame grayscale image, and u12 represents the minimum grayscale value contained in the corresponding current frame grayscale image. It is represented by the probability of the pixel with gray value i appearing in the corresponding current frame gray image, u21 is represented by the maximum gray value contained in the corresponding predicted frame gray image, and u22 is represented by the minimum gray value contained in the corresponding predicted frame gray image. It is expressed as the probability of the pixel with gray value i appearing in the corresponding predicted frame gray image; The specific formula of the characteristic point difference value TC is: ， ， Wherein, k1 represents the first preset weight, k2 represents the second preset weight, TS1 represents the number of feature points of the corresponding current frame grayscale image, TS2 represents the number of feature points of the corresponding predicted frame grayscale image, and FTS represents the number of non-matching feature points in the predicted frame grayscale image relative to the corresponding current frame grayscale image; The specific formula for predicting the reliable value of gaze is: ， Wherein, M represents the number of adjacent segmented regions corresponding to the segmented region in the predicted frame grayscale image, and N represents the number of pixels in the corresponding segmented region in the predicted frame grayscale image. It represents the number of pixels in the mth adjacent segmentation area of ​​the corresponding segmentation area in the predicted frame grayscale image, e is a natural constant, Represents the mean grayscale value of the corresponding segmented area in the predicted frame grayscale image, It represents the mean gray value of the mth adjacent segmentation area corresponding to the segmentation area in the gray image of the predicted frame. Represents the information entropy of the corresponding segmented area in the predicted frame grayscale image, Represents the information entropy of the mth adjacent segmented area corresponding to the segmented area in the predicted frame grayscale image.

6. The VR somatosensory detection optimization system according to claim 5, characterized in that: The step of judging whether the predicted frame grayscale image is a similar image to the corresponding current frame grayscale image according to the information change value and the feature point difference value is as follows: If the information change value is less than the preset change threshold and the feature point difference value is less than the preset difference threshold, the corresponding predicted frame grayscale image belongs to a similar image of the corresponding current frame grayscale image; The method of judging the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image according to the information change value and the feature point difference value is specifically: If the information change value is greater than or equal to the preset change threshold and the feature point difference value is less than the preset difference threshold, or the information change value is less than the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, then the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is an obvious local change; If the information change value is greater than or equal to the preset change threshold and the feature point difference value is greater than or equal to the preset difference threshold, the change type of the predicted frame grayscale image relative to the corresponding current frame grayscale image is a global change.

Images