A gaze point prediction method and system for VR large space immersive tour

Abstract

The application belongs to the field of artificial intelligence, computer vision and computer graphics, and discloses a gaze point prediction method and system for VR large space immersive tour, which comprises the following steps: acquiring an eye image transmitted by a near-eye camera; constructing a gaze point prediction network model; inputting the eye image into the gaze point prediction network model to output a predicted line-of-sight direction. The application provides a method for accurately predicting a gaze point in real time, and through the method of eliminating 80% irrelevant pixels in the input image and the method of outputting a prediction value by a multi-level neural network, the calculation complexity is reduced in multiple dimensions, thereby reducing the system delay and improving the rendering quality.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence, computer vision and computer graphics, and particularly relates to a gaze point prediction method and system for VR large-space immersive tour. BACKGROUND

[0002] With the development of meta universe, smart tourism and smart city, the demand for immersive rendering in the consumer market is increasing. Using head-mounted devices, especially VR devices, to render large-scale scenes with high resolution and high frame rate has gradually become a basic requirement for immersive experience, and is widely used in LBE (LBE (Location-Based Entertainment) is an immersive experience mode realized through VR technology, mainly applied in larger physical spaces to provide rich interactive experiences) large-space immersive offline experience halls, such as rendering a virtual scene with 100 million triangles at 4K resolution (Apple Vision pro supports 3860x3200 super 4K resolution) and 60FPS (Frames Per Second, i.e. frames per second. Generally, real-time rendering requires at least 30FPS, and VR requires at least 60FPS, preferably 120FPS).

[0003] However, compared with a desktop computer with a high-performance independent graphics card, the hardware performance of a head-mounted (head-mounted display) device is relatively low. Therefore, generally, real-time eye tracking technology is used to determine the direction and position of the user's gaze, and then high-resolution graphics images are rendered onto the fovea (foveal), which is the highest vision area on the retina, while the peripheral area is presented with lower resolution, thereby improving rendering efficiency and meeting the operating requirements of the head-mounted device to some extent.

[0004] However, the existing eye tracking and gaze point prediction methods have low accuracy, especially at the 95th percentile, which is relatively large, resulting in the rendering system being unable to accurately adapt to the user's visual focus, thereby causing misalignment between the rendering details and the real visual line position, leading to obvious artifacts and reducing the user's experience. In order to not affect the user experience and ensure visual quality, the area and resolution of the fovea can only be expanded to compensate for the inaccuracy of eye tracking, but this will increase the cost of calculation and rendering, increase the delay, cause the system delay to exceed 16.67ms, thereby failing to maintain a frame rate of 60FPS, resulting in head-mounted rendering lag and unsmooth problems. SUMMARY

[0005] To solve the problems in the prior art, the application provides a gaze point prediction method and system for VR large-space immersive tour, which provides a method for predicting a gaze point in real time and accurately, and reduces the calculation complexity in multiple dimensions by removing 80% of irrelevant pixels in an input image and using a neural network multi-level output prediction value method, thereby reducing system delay and improving rendering quality.

[0006] To achieve the above object, the application provides the following scheme:

[0007] A gaze point prediction method for VR large-space immersive tour, the method comprising:

[0008] obtaining an eye image transmitted by a near-eye camera;

[0009] constructing a gaze point prediction network model;

[0010] inputting the eye image into the gaze point prediction network model to output a predicted line-of-sight direction.

[0011] Preferably, inputting the eye image into the gaze point prediction network model to output a predicted line-of-sight direction comprises:

[0012] aligning the eye image and performing cropping;

[0013] delivering the cropped image to a convolution layer to perform shallow feature extraction;

[0014] after processing, passing to a first attention module, a second attention module, a third attention module and a fourth attention module to perform deep feature extraction, and outputting a predicted line-of-sight direction.

[0015] Preferably, the step of cropping the aligned eye image comprises:

[0016] based on prior knowledge and a preset threshold, performing an inverse binary mask operation on the eye image to obtain a mask image;

[0017] performing a morphological opening operation on the mask image to obtain a connected body image;

[0018] according to the prior that the pixel density of the pupil region is higher than that of other regions, searching for a maximum connected body region in the connected body image, and then according to the prior that the pupil is circular or elliptical, calculating the circular arc of the maximum connected body region;

[0019] based on the circular arc of the maximum connected body region, if the maximum connected body region is approximately circular or elliptical, it indicates that the maximum connected body region is the region where the pupil is located, and the center of the maximum connected body region is taken as the center of the pupil;

[0020] Create a rectangle with the pupil center as the rectangle center.

[0021] Preferably, creating a rectangle with the pupil center as the rectangle center comprises:

[0022] According to the value range of the pupil outer diameter, the value range of the iris outer diameter, and the value range of the sclera outer diameter, the longest outer diameter of the maximum connected body region is calculated.

[0023] Based on the horizontal 10 times and the vertical 6 times of the longest outer diameter, a rectangle is obtained as a clipping region.

[0024] If the rectangle boundary exceeds the image boundary, the clipping region is translated to the inside of the image until the clipping frame coincides with the image boundary, so that the clipping region is within the image range.

[0025] The application also provides a gaze point prediction system for VR large space immersive tour, which is used to realize any one of the methods, and comprises an acquisition module, a construction module, and a prediction module.

[0026] The acquisition module is used to acquire an eye image transmitted by a near-eye camera.

[0027] The construction module is used to construct a gaze point prediction network model.

[0028] The prediction module is used to input the eye image into the gaze point prediction network model and output a predicted line of sight direction.

[0029] Preferably, the prediction module comprises a clipping unit, a first extraction unit, and a second extraction unit.

[0030] The clipping unit is used to align and clip the eye image.

[0031] The first extraction unit is used to pass the clipped image to a convolution layer for shallow feature extraction.

[0032] The second extraction unit is used to pass the processed image to a first attention module, a second attention module, a third attention module, and a fourth attention module for deep feature extraction, and output a predicted line of sight direction.

[0033] Preferably, the clipping unit comprises a mask subunit, an operation subunit, a calculation subunit, a judgment subunit, and a creation subunit.

[0034] The mask subunit is used to perform an inverse binary mask operation on the eye image based on prior knowledge and a preset threshold to obtain a mask image.

[0035] The operator unit is used for performing a morphological opening operation on the mask image to obtain a connected body image;

[0036] The calculation sub-unit is used for searching a maximum connected body region in the connected body image according to the priori that the pixel density of the pupil region is higher than that of other regions, and calculating a circular arc of the maximum connected body region according to the priori that the pupil is circular or elliptical.

[0037] The judgment sub-unit is used for judging whether the maximum connected body region is the pupil region based on the circular arc of the maximum connected body region, and taking the center of the maximum connected body region as the center of the pupil if the maximum connected body region is approximately circular or elliptical.

[0038] The creation sub-unit is used for creating a rectangle with the center of the pupil as the center of the rectangle.

[0039] Preferably, the step of creating a rectangle with the center of the pupil as the center of the rectangle comprises:

[0040] The longest outer diameter of the maximum connected body region is calculated according to the value range of the outer diameter of the pupil, the value range of the outer diameter of the iris and the value range of the outer diameter of the sclera.

[0041] A rectangle is obtained as a clipping region based on 10 times the horizontal direction and 6 times the vertical direction of the longest outer diameter.

[0042] If the boundary of the rectangle exceeds the image boundary, the clipping region is translated to the inside of the image until the clipping frame coincides with the image boundary, so that the clipping region is within the image range.

[0043] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any one of the application when executing the program.

[0044] The application further provides a computer readable storage medium storing a computer program, wherein the computer program is executable to implement the method of any one of the application.

[0045] Compared with the prior art, the application has the following beneficial effects:

[0046] The application provides a method for accurately predicting a gaze point in real time, and reduces the calculation complexity in multiple dimensions by removing 80% of irrelevant pixels in an input image and using a neural network to output a predicted value, thereby reducing system delay and improving rendering quality.

[0047] The application adaptively adjusts the speed of prediction to meet different requirements for real-time performance and accuracy, and the gaze point prediction is accurate. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the present application, the following briefly introduces the drawings needed to be used in the embodiments. Obviously, the drawings described in the following embodiments are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0049] Figure 1 An eye movement tracking schematic diagram for the embodiment of the present application;

[0050] Figure 2 A gaze point prediction method flowchart for the embodiment of the present application of VR large space immersive tour;

[0051] Figure 3 A structural schematic diagram of the electronic device of the embodiment of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0053] 1010, processor; 1020, memory; 1030, input / output interface; 1040, communication interface; 1050, bus. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the scope of protection of the present application.

[0055] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0056] Firstly, some technical terms used in the present application are described:

[0057] 1. Eye movement tracking (gaze point prediction) error

[0058] (1) When the gaze point is located at the center of the view, the radius (in pixels) r of the foveal region f is as follows (as shown in Figure 1 ):

[0059] r f = r i + e = p d tan (theta i + delta theta)

[0060] Where p represents the pixel density of the display, d represents the distance between the gaze point and the screen, theta ieccentricity angle of fovea, θ f = θ i + δθ represents the eccentricity angle of fovea considering the gaze point prediction error δθ. The foveal region radius without tracking error is r i = ρd· tan(θ i ), and the error constant e = ρd· (tan(θ f ) - tan(θ i )) is used to represent the change of foveal region radius.

[0061] (2) If the gaze point is not at the center of the view, its actual radius is smaller than r f , and it can still be rendered as r f , and the error is much smaller than the gaze point prediction error δθ, so this error can be ignored.

[0062] (3) Generally, θ i = 5.2°, and the internal foveal region is set to 15°. This is equivalent to dividing the screen into three regions with two concentric circles: the inner circle is the foveal region, which is drawn with the highest resolution; the outer circle is the internal foveal region, which is drawn with 1 / 4 resolution; and the area inside the screen outside the outer circle is drawn with 1 / 16 resolution.

[0063] 2. Gaze point rendering pipeline

[0064] (1) The gaze point rendering pipeline of modern head-mounted devices generally includes three main components: a near-eye camera, an interconnection link (used to link the near-eye camera and the host processor, such as MIPI C-PHY, which is a physical layer standard developed by the MIPI Alliance for high-speed serial data transmission), and a host processor.

[0065] (2) Gaze point rendering pipeline:

[0066] 1) Capture eye images using a near-eye monochrome camera

[0067] 2) Preprocess and read out the images through an image signal processor (ISP)

[0068] 3) Then connect and transmit the images to the host processor through MIPI C-PHY

[0069] 4) After the host processor receives the images, it first sends them to the gaze point prediction network

[0070] 5) The gaze point prediction network estimates the line-of-sight direction

[0071] 6) Use the estimated line-of-sight direction to guide the host processor in the foveal rendering process.

[0072] 3. System latency

[0073] (1) Camera perception and shooting roughly need 1ms, MIPI C-PHY communication is less than 1ms, and the existing gaze point prediction network (line of sight direction detection) is about 2m to 20ms, and rendering and display is about 2m or more.

[0074] (2) The formula is: t total = t sensing + t comm + t tracking + t fr , wherein t sensing represents the camera perception delay, t comm represents the MIPI C-PHY communication delay, t tracking represents the gaze point prediction delay, and t fr represents the gaze point rendering delay. The method optimizes the DNN network of the gaze point prediction, thereby reducing t tracking .

[0075] (3) The greater the delay, the lower the frame rate. Generally, the head-mounted device requires a frame rate of at least 60FPS. If the frame rate is too low, it will cause lag and dizziness.

[0076] According to the approximate distribution of the gaze point rendering delay, the focus of the method is to optimize the gaze point prediction network and the rendering process, so as to control the system delay to be less than 16.67ms, that is, to achieve real-time rendering of 60FPS.

[0077] Embodiment one

[0078] As shown in Figure 2 , the embodiment of the application provides a gaze point prediction method for VR large space immersive tour, which comprises:

[0079] obtaining an eye image transmitted by a near-eye camera;

[0080] constructing a gaze point prediction network model;

[0081] inputting the eye image into the gaze point prediction network model, and outputting a predicted line of sight direction.

[0082] In this embodiment, when the "gaze point prediction network" receives the eye image transmitted by the near-eye camera, the image is first aligned and cropped, and then the cropped image is transmitted to the "convolution layer" for shallow feature extraction. After processing, it is transmitted to the "first attention module", "second attention module", "third attention module" and "fourth attention module" in turn for deep feature extraction, and outputs the predicted line of sight direction, so as to meet the two requirements of real-time and high quality.

[0083] In this embodiment, the input image: the eye image captured by the near-eye camera. The output: the predicted gaze direction (a two-dimensional value, representing the direction vector).

[0084] In this embodiment, (1) the purpose of cropping: because the eye image captured by the near-eye camera contains redundant pixels irrelevant to the gaze point prediction task, such as background, facial muscles, etc., these pixels have a negative impact on the prediction result and increase the calculation cost, especially on the head-mounted device, the increase in calculation overhead will affect the delay and frame rate, so the redundant pixels are removed first.

[0085] (2) Cropping algorithm:

[0086] 1) Algorithm principle: first find the pupil, then take the pupil as the center of the rectangle, and most of the irrelevant pixels can be cropped. Because the pixels around the pupil are the most effective inputs for the gaze point prediction task.

[0087] 2) First, use the priori that the pupil is usually darker than the surrounding sclera and iris, set a threshold, and perform an inverse binary mask operation on the input image to get a mask image.

[0088] 3) Perform morphological opening on the above mask image (morphological opening is a combination process of erosion operation and dilation operation. The erosion operation can remove small objects or details in the image, while the dilation operation can restore the image structure that has been eroded to some extent. Through the combination of the two operations, the opening operation can achieve the effect of removing noise, separating connected objects, and smoothing the boundaries of larger objects, etc.), to reduce the noise in the image, so that only the pixels in the pupil and eyelash regions with darker colors will be retained. Through morphological opening, a connected component map is obtained.

[0089] 4) According to the priori that the pixel density of the pupil region is higher than that of other regions, search for the largest connected component region in the connected component map, and according to the priori that the pupil is circular or elliptical, calculate the circular arc of the largest connected component region, the formula is:

[0090]

[0091] Where A represents the area of the connected component, P represents the boundary perimeter of the connected component, and when C approaches 1, it indicates that the region is close to circular. A threshold value can be set When At this time, it is shown that the connected body is circular or elliptical (may make The area and perimeter of the connected body can be calculated using the regionprops function in MATLAB software. Other algorithms can also be used.

[0092] 5) If the connected body is approximately circular or elliptical, it indicates that the connected body region is the region where the pupil is located, and the center of this region can be taken as the center of the pupil. At this time, the position of the pupil center has been found through the algorithm. If the largest connected body region is not approximately circular or elliptical, it indicates that the image quality is not high, and it is directly discarded, that is, the next image is processed.

[0093] 6) Finally, a rectangle is created with the pupil center as the center of the rectangle

[0094] According to prior knowledge: the value range of the outer diameter of the pupil is 2.5-4mm, the value range of the outer diameter of the iris is about 11.5mm, and the value range of the outer diameter of the sclera is about 23.5mm. Therefore, the longest outer diameter of the above largest connected body is calculated first, and then a rectangle is obtained as the cropping region according to 10 times the length in the horizontal direction and 6 times the length in the vertical direction.

[0095] If the boundary of this rectangle exceeds the image boundary, the cropping region is translated towards the inside of the image until the cropping frame coincides with the image boundary to ensure that the cropping region is within the image range.

[0096] Generally, the width of the cropping region can be reduced to 1 / 4 of the original, the height can be reduced to 1 / 5 of the original, and the total area can be reduced to 1 / 20 of the original, effectively reducing the range by 95%, greatly reducing the image size input to the subsequent network, which is conducive to greatly reducing the delay, improving the computing performance, and meeting the real-time computing requirements.

[0097] In this embodiment, there are 5 convolutional layers in the architecture diagram, but 3 attention modules are not drawn, and each module is iterated twice, so if all are counted, there are 12 in total.

[0098] (1) All convolutional layers in this method have the same structure, which is composed of a residual network, and have the same effect, which is to downsample and reduce the feature map size by half and increase the channel number by 4 times. Because of the existence of the residual network, the channel is 2 times the original after the first iteration, and 4 times the original after the second iteration.

[0099] (2) I con,1 In addition to downsampling, there is also a function of shallow feature extraction.

[0100] (3) The input image The first convolutional layer outputs

[0101] Where H, W are the height and width of the input image size, C is the number of channels, generally C = 3, that is, RGB three channels.

[0102] (4) The second convolutional layer is slightly more complex. The second convolutional layer is inside the "first attention module". Because the "first attention module" is iterated twice, in the first iteration, in the second iteration,

[0103] (5) The third convolutional layer The fourth convolutional layer The fifth convolutional layer

[0104] In this embodiment, (1) the linear layer is also called a fully connected layer, and its main function is to perform linear transformation on the input data. The line-of-sight direction is directly output from the linear layer.

[0105] (2) Each "attention module" has a linear layer, so it will output a predicted line-of-sight direction; and because each "attention module" is iterated twice, each "attention module" will ultimately output two predicted line-of-sight directions.

[0106] (3) Not every layer of attention module predicts the line-of-sight direction, but only the "first attention module" outputs one, in order to meet the requirement of real-time; and the last one is output, in order to more accurately predict the line-of-sight method.

[0107] (4) Specifically, a time threshold ε is set. When the final "second line-of-sight direction" is not generated within this time threshold (may be due to the complexity of the current scene, network delay, etc.), the "first line-of-sight direction" is directly used. It can also be set by user preference, such as the user needs low delay, then reduce the time threshold; if the user needs high quality, then increase the time threshold, or even directly use the "second line-of-sight direction".

[0108] (5) There is another case. If the "second line-of-sight direction" is generated before the next prediction, it needs to be determined whether the "first line-of-sight direction" and the "second line-of-sight direction" are in the central foveal region. If they are, the "first line-of-sight direction" is still processed; if they are not, it means that the deviation is relatively large, and the "second line-of-sight direction" needs to be redrawn. If the next prediction result comes out before the redrawing is completed, the line-of-sight direction of the next prediction is used for drawing, ignoring this time of redrawing.

[0109] (6) In addition, a skip connection is added in the architecture diagram, which connects the output of the "first attention module" to the input of the "fourth attention module". A line is drawn in the middle, which is similar to the U-Net. The purpose is to fuse the features extracted by the shallow layer and the features extracted by the deep layer, so that the prediction result can consider both local information and global information, thereby obtaining more accurate prediction results.

[0110] In this embodiment, (1) the "first attention module", "second attention module", "third attention module", and "fourth attention module" have the same structure. The reason for adding four and iterating each one is to better learn deep image features. By inserting a convolution layer, the size of the feature map is reduced and the number of channels is increased, thereby actually increasing the receptive field of the convolution kernel, so that more global information can be learned.

[0111] (3) The most core of the "first attention module" is the "mixed attention module". This structure is basically similar to the standard Transformer, but there are two differences.

[0112] 1) The first difference is that in the multi-head attention module of the standard Transformer, we use a channel attention module in parallel (without limiting the specific implementation of this channel attention module, such as the Squeeze-and-Excitation (SE) module, the Efficient Channel Attention (ECA) module, and variants of the two).

[0113] 2) The second difference is that we use a mechanism of moving three small windows of different sizes on the feature map (in principle, it is inspired by the window moving mechanism of Swin Transformer). The sizes of the three small windows are and Padding is used to fill the surrounding of the image (only when the image size and the small window size are not multiples of each other will padding be done), and then cross-attention is calculated inside the small window. In order to solve the problem of pixel connectivity between adjacent small windows, we use three small windows, which are gradually increased in size, thereby increasing the receptive field, so that the pixels originally in two small windows can also be inside a larger window, thereby calculating cross-attention.

[0114] (4) The symbol represents feature fusion, and the fusion method is to keep the size unchanged and double the number of channels.

[0115] (5) The "first attention module" is iterated twice. In the second iteration, the symbol The output feature map is taken as input into the "mixed attention module" and in the skip connection as the next input of feature fusion. Similarly, the other attention modules are also iterated twice respectively.

[0116] In this embodiment, (1) the average gaze point prediction error

[0117]

[0118] where B represents a batch set of training data, represents the training data set in batch b, y d and represent the predicted and true gaze direction of the training data d respectively.

[0119] (2) In order to solve the problem that the 95th percentile of the gaze point error is large, we use the maximum tracking error to solve this problem

[0120]

[0121] where B represents a batch set of training data, represents the training data set in batch b, θ d and represent the predicted gaze direction (radian) and true gaze direction (radian) of the input sample d in the training data set respectively, and N is a scaling factor that controls the approximation degree. In the training process, the value of N is carefully adjusted to adapt to the value distribution of the input training data, so as to ensure better convergence of the training process.

[0122] (3) Total loss function

[0123] (4) In order to obtain better training effect, we take different training strategies:

[0124] Let the first gaze direction be trained faster to meet the 95% accuracy, and use to train the "first gaze direction" subnetwork to quickly predict the result.

[0125] Use to train the entire network, i.e. "second gaze direction", and the corresponding λ = 0.8, so that plays a greater role, thereby more accurately predicting the result and solving the problem of more than 95% accuracy.

[0126] Embodiment two

[0127] The application further provides a gaze point prediction system for VR large-space immersive tour, which is used for realizing the method in any one of the above-mentioned embodiments.

[0128] The acquisition module is configured to acquire an eye image transmitted by a near-eye camera.

[0129] The construction module is configured to construct a gaze point prediction network model.

[0130] The prediction module is configured to input the eye image into the gaze point prediction network model and output a predicted line-of-sight direction.

[0131] In this embodiment, the prediction module comprises a clipping unit, a first extraction unit and a second extraction unit.

[0132] The clipping unit is configured to align and clip the eye image.

[0133] The first extraction unit is configured to pass the clipped image to a convolutional layer for shallow feature extraction.

[0134] The second extraction unit is configured to pass the processed image to a first attention module, a second attention module, a third attention module and a fourth attention module for deep feature extraction and output a predicted line-of-sight direction.

[0135] In this embodiment, the clipping unit comprises a mask subunit, an operation subunit, a calculation subunit, a judgment subunit and a creation subunit.

[0136] The mask subunit is configured to perform an inverse binary mask operation on the eye image based on prior knowledge and a preset threshold to obtain a mask image.

[0137] The operation subunit is configured to perform a morphological opening operation on the mask image to obtain a connected body image.

[0138] The calculation subunit is configured to search for a maximum connected body region in the connected body image according to the prior that the pixel density of the pupil region is higher than that of other regions, and then calculate the circular arc of the maximum connected body region according to the prior that the pupil is circular or elliptical.

[0139] The judgment subunit is configured to, based on the circular arc of the maximum connected body region, if the maximum connected body region is approximately circular or elliptical, it is indicated that the maximum connected body region is the pupil region, and the center of the maximum connected body region is taken as the center of the pupil.

[0140] The creation subunit is configured to create a rectangle with the center of the pupil as the center of the rectangle.

[0141] In the embodiment, a rectangle is created with the pupil center as the center of the rectangle, including:

[0142] According to the value range of the pupil outer diameter, the value range of the iris outer diameter, and the value range of the sclera outer diameter, the longest outer diameter of the largest connected region is calculated.

[0143] Based on 10 times the horizontal direction and 6 times the vertical direction of the longest outer diameter, a rectangle is obtained as a clipping region.

[0144] If the rectangle boundary exceeds the image boundary, the clipping region is translated to the inside of the image until the clipping frame coincides with the image boundary, so that the clipping region is within the image range.

[0145] Embodiment three

[0146] Based on the same inventive concept, the disclosure also provides an electronic device corresponding to any of the above embodiment methods, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to realize the three-dimensional structure recovery method of the high-quality urban renewal landscape building according to any one of the above embodiments.

[0147] Figure 3 A more specific hardware structure of an electronic device provided by the embodiment is shown, which can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for communication within the device.

[0148] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit, central processor), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present specification.

[0149] The memory 1020 can be implemented by a ROM (Read Only Memory, read-only memory), a RAM (Random Access Memory, random access memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 1020 and executed by the processor 1010.

[0150] The input / output interface 1030 is configured to connect an input / output module to realize information input and output. The input / output module can be configured in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0151] The communication interface 1040 is configured to connect a communication module (not shown in the figure) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired manner (such as a USB (Universal Serial Bus), a network cable, etc.) or a wireless manner (such as a mobile network, WIFI (Wireless Fidelity), Bluetooth, etc.).

[0152] The bus 1050 includes a channel to transmit information between various components (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0153] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only include components necessary for the implementation of the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.

[0154] The system of the above embodiments is used to implement the three-dimensional structure recovery method of a high-quality urban renewal landscape building in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0155] Embodiment Four

[0156] Based on the same inventive concept, the disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the three-dimensional structure recovery method of a high-quality urban renewal landscape building according to any of the above embodiments.

[0157] The computer readable media of the present embodiments includes permanent and non-permanent, removable and non-removable media can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device.

[0158] The storage medium of the above embodiments stores computer instructions for causing the computer to perform a three-dimensional structure recovery method of a high-quality urban renewal landscape building as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here.

[0159] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present disclosure (including claims) is limited to these examples; under the idea of the present disclosure, the above embodiments or technical features between different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes of different aspects of the embodiments of the present disclosure as described above. In order to be brief, they are not provided in detail.

[0160] In addition, in order to simplify the description and discussion, and so as not to make the embodiments of the present disclosure difficult to understand, the well-known power / ground connections of integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. In addition, the devices can be shown in the form of block diagrams in order to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented the embodiments of the present disclosure (i.e. these details should be fully within the understanding of those skilled in the art). Where specific details (e.g. circuits) are set forth in order to describe an exemplary embodiment of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with variations on these specific details. Therefore, these descriptions should be considered illustrative rather than limiting.

[0161] Although the present disclosure has been described in connection with certain embodiments, numerous modifications and alterations mentioned above are well within the skills of those in the art. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0162] Therefore, the units of each example described in the embodiments of the present application can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0163] The above-described embodiments are only a description of the preferred mode of the present application, and are not intended to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope determined by the claims of the present application.

Claims

1. A gaze point prediction method for VR large space immersive tour, characterized in that, The method comprises: obtaining an eye image transmitted by a near-eye camera; constructing a gaze point prediction network model; inputting the eye image into the gaze point prediction network model to output a predicted line of sight direction; inputting the eye image into the gaze point prediction network model to output a predicted line of sight direction comprises: aligning the eye image and performing cropping; delivering the cropped image to a convolutional layer for shallow feature extraction; after processing, passing back to a first attention module, a second attention module, a third attention module, and a fourth attention module for deep feature extraction, and outputting a predicted line of sight direction; The first attention module, the second attention module, the third attention module, and the fourth attention module have the same structure, each of which iterates once to learn deep image features and learn more global information through an inserted convolutional layer; The most core part of the first attention module is a hybrid attention module, which is improved based on a standard Transformer: In the part of the multi-head attention module of the standard Transformer, a channel attention module is used in parallel, including an SE module, an ECA module, and variants of the two; The mechanism of moving the feature map with three different sizes of small windows, three small window sizes are and and padding is used to fill the image around, and then calculate the cross attention inside the small window; Symbols denotes feature map fusion, and the fusion manner is that the size is unchanged and the channel number is doubled; The first attention module iterates 2 times, and the second time, the symbol The output feature map is input into the hybrid attention module as input, and in the skip connection, as the next input of feature fusion, and similarly, the other attention modules also iterate twice respectively.

2. The method of claim 1, wherein, The cropping of the aligned eye image comprises: based on prior knowledge and a preset threshold, performing an inverse binary mask operation on the eye image to obtain a mask image; performing a morphological opening operation on the mask image to obtain a connected body image; According to the priori that the pixel density of the pupil region is higher than that of other regions, the maximum connected body region in the connected body image is searched, and then the circular arc of the maximum connected body region is calculated according to the priori that the pupil is circular or elliptical; Based on the circular arc of the maximum connected body region, if the maximum connected body region is approximately circular or elliptical, it indicates that the maximum connected body region is the pupil region, and the center of the maximum connected body region is taken as the center of the pupil; Taking the pupil center as the center of the rectangle, a rectangle is created.

3. The method of claim 2, wherein, Taking the pupil center as the center of the rectangle, a rectangle is created, which comprises: According to the value range of the pupil outer diameter, the value range of the iris outer diameter, and the value range of the sclera outer diameter, the longest outer diameter of the maximum connected body region is calculated; Based on the horizontal 10 times and the vertical 6 times of the longest outer diameter, a rectangle is obtained as a cropping region; If the rectangle boundary exceeds the image boundary, the cropping region is translated inward until the cropping frame coincides with the image boundary, so that the cropping region is within the image range.

4. A gaze point prediction system for VR large space immersive tours, the system being configured to implement the method of any one of claims 1-3, characterized in that, The system comprises an acquisition module, a construction module, and a prediction module. The acquisition module is configured to acquire an eye image transmitted by a near-eye camera. The construction module is configured to construct a gaze point prediction network model. The prediction module is configured to input the eye image into the gaze point prediction network model to output a predicted line of sight direction.

5. The system of claim 4, wherein, The prediction module comprises a cropping unit, a first extraction unit, and a second extraction unit. The cropping unit is configured to align the eye image and perform cropping. The first extraction unit is configured to deliver the cropped image to a convolutional layer for shallow feature extraction. The second extraction unit is configured to process the backward direction and pass the processed result to the first attention module, the second attention module, the third attention module and the fourth attention module for deep feature extraction and output of the predicted gaze direction.

6. The system of claim 5, wherein, The cropping unit comprises a mask subunit, an operation subunit, a calculation subunit, a judgment subunit and a creation subunit. The mask subunit is configured to perform an inverse binary mask operation on the eye image based on prior knowledge and a preset threshold to obtain a mask image. The operation subunit is configured to perform a morphological opening operation on the mask image to obtain a connected body image. The calculation subunit is configured to search for a maximum connected body region in the connected body image according to the prior that the pixel density of the pupil region is higher than that of other regions, and calculate the circular arc of the maximum connected body region according to the prior that the pupil is circular or elliptical. The judgment subunit is configured to determine, based on the circular arc of the maximum connected body region, whether the maximum connected body region is approximately circular or elliptical, and if so, determine that the maximum connected body region is the pupil region and take the center of the maximum connected body region as the center of the pupil. The creation subunit is configured to create a rectangle with the center of the pupil as the center of the rectangle.

7. The system of claim 6, wherein, The creation of the rectangle with the center of the pupil as the center of the rectangle comprises: calculating the longest outer diameter of the maximum connected body region according to the value range of the outer diameter of the pupil, the value range of the outer diameter of the iris and the value range of the outer diameter of the sclera; obtaining a rectangle as a cropping region based on 10 times the horizontal direction and 6 times the vertical direction of the longest outer diameter; and if the boundary of the rectangle exceeds the boundary of the image, moving the cropping region inward until the boundary of the cropping region coincides with the boundary of the image.

8. An electronic device, comprising: The computer readable storage medium stores a computer program, and when the computer program is executed, the method of any one of claims 1 to 3 is implemented.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program is executed, the method of any one of claims 1 to 3 is implemented.

Images