Method and system for predicting fixation point of VR large-space immersive sightseeing

Abstract

The invention belongs to the field of artificial intelligence, computer vision and computer graphics, and discloses a fixation point prediction method and system for VR large-space immersive sightseeing, and the method comprises the steps: obtaining an eye image transmitted by a near-eye camera; constructing a fixation point prediction network model; and inputting the eye image into the fixation point prediction network model, and outputting a predicted sight line direction. The invention provides a method for accurately predicting the fixation point in real time, and the calculation complexity is reduced in multiple dimensions by eliminating 80% of irrelevant pixels in the input image and outputting the predicted value in multiple levels by the neural network, so that the system delay is reduced, and the rendering quality is improved.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence, computer vision, and computer graphics, and particularly relates to a method and system for predicting fixation points in a large-space immersive VR tour. Background Art

[0002] With the development of the metaverse, smart cultural tourism, and smart cities, the consumer market has an increasing demand for immersive rendering. Using a head-mounted display device, especially a VR device, to render large-scale scenes at high resolution and high frame rate has gradually become a basic requirement for immersive experiences, and is widely used in large-space immersive offline experience halls for LBE (Location-Based Entertainment, an immersive experience method achieved through VR technology, mainly applied in a relatively large physical space to provide rich interactive experiences). For example, rendering a virtual scene with 100 million triangles at a resolution of 4K (the Apple Vision Pro supports a super 4K resolution of 3860x3200) and 60 FPS (Frames Per Second. Generally, real-time rendering requires at least 30 FPS, VR requires at least 60 FPS, and preferably 120 FPS).

[0003] However, compared with a desktop computer with a high-performance independent graphics card, the hardware performance of a head-mounted display (HMD) device is relatively low. Therefore, generally, real-time eye tracking technology is used to determine the direction and position that the user is looking at, and then high-resolution graphics images are rendered onto the fovea (the fovea, also known as the foveola centralis, is the central concave part of the macula area of the retina and is the area of the retina with the most sensitive vision (color discrimination ability and resolution ability). Approximately 3.5 mm temporal to the optic disc, there is a yellowish area called the macula, and the central depression therein is the fovea. It appears as a light yellow circular spot with a diameter of about 1.5 mm) area with the highest visual acuity in the retina, and the surrounding area is presented at a lower resolution, thereby improving the rendering efficiency and meeting the operating requirements of the HMD device to a certain extent.

[0004] However, the accuracy of existing eye tracking and fixation point prediction methods is not high, especially the error at the 95th percentile is relatively large, resulting in the rendering system being unable to accurately adapt to the user's visual focus, thus causing an alignment problem between the rendering details and the actual line-of-sight position, leading to obvious artifacts and reducing the user experience. To avoid affecting the user experience and ensure visual quality, only the area and resolution of the fovea can be enlarged to compensate for inaccurate eye tracking, but this will increase the computational and rendering costs and increase the latency, resulting in the system latency exceeding 16.67 ms, thus being unable to maintain a frame rate of 60 FPS and causing problems such as stuttering and unsmooth rendering of the HMD. Summary of the Invention

[0005] To solve the problems existing in the prior art, the present invention provides a method and system for predicting a fixation point in a large-space immersive VR tour. This method provides a method for accurately predicting the fixation point in real time, and reduces the computational complexity in multiple dimensions by removing 80% of the irrelevant pixels in the input image and the method of multi-level output prediction values of the neural network, thereby reducing the system latency and improving the rendering quality.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A method for predicting a fixation point in a large-space immersive VR tour, the method comprising:

[0008] Obtain an eye image transmitted from a near-eye camera;

[0009] Construct a fixation point prediction network model;

[0010] Input the eye image into the fixation point prediction network model and output the predicted line of sight direction.

[0011] Preferably, inputting the eye image into the fixation point prediction network model and outputting the predicted line of sight direction includes:

[0012] Align the eye image and perform cropping;

[0013] Transmit the cropped image to the convolutional layer for shallow feature extraction;

[0014] After processing, pass it backward to the first attention module, the second attention module, the third attention module, and the fourth attention module for deep feature extraction, and output the predicted line of sight direction.

[0015] Preferably, cropping the aligned eye image includes:

[0016] Based on prior knowledge and a preset threshold, perform an inverse binary mask operation on the eye image to obtain a mask image;

[0017] Perform a morphological opening operation on the mask image to obtain a connected component image;

[0018] According to the prior that the pixel density in the pupil area is higher than that in other areas, search for the largest connected component area in the connected component image, and then calculate the circular arc degree of the largest connected component area according to the prior that the pupil is circular or elliptical;

[0019] Based on the circular arc degree of the largest connected component area, if the largest connected component area is approximately circular or elliptical, it indicates that the largest connected component area is the area where the pupil is located, and take the center of the largest connected component area as the center of the pupil;

[0020] Create a rectangle with the pupil center as the center of the rectangle.

[0021] Preferably, creating a rectangle with the pupil center as the center of the rectangle includes:

[0022] According to the value range of the pupil outer diameter, the iris outer diameter, and the sclera outer diameter, calculate the longest outer diameter of the largest connected component region;

[0023] Based on 10 times the horizontal and 6 times the vertical of the longest outer diameter, obtain a rectangle as the cropping region;

[0024] If the rectangle boundary exceeds the image boundary, translate the cropping region inward until the cropping border coincides with the image boundary, so that the cropping region is within the image range.

[0025] The present invention also provides a fixation point prediction system for VR large-space immersive tours, which is used to implement any one of the above methods. The system includes: an acquisition module, a construction module, and a prediction module;

[0026] The acquisition module is used to acquire the eye image transmitted from the near-eye camera;

[0027] The construction module is used to construct a fixation point prediction network model;

[0028] The prediction module is used to input the eye image into the fixation point prediction network model and output the predicted line of sight direction.

[0029] Preferably, the prediction module includes: a cropping unit, a first extraction unit, and a second extraction unit;

[0030] The cropping unit is used to align and crop the eye image;

[0031] The first extraction unit is used to transfer the cropped image to the convolutional layer for shallow feature extraction;

[0032] The second extraction unit is used to transfer the processed image backward to the first attention module, the second attention module, the third attention module, and the fourth attention module for deep feature extraction, and output the predicted line of sight direction.

[0033] Preferably, the cropping unit includes: a mask sub-unit, an operation sub-unit, a calculation sub-unit, a judgment sub-unit, and a creation sub-unit;

[0034] The mask sub-unit is used to perform an inverse binary mask operation on the eye image based on prior knowledge and a preset threshold to obtain a mask image;

[0035] The operator unit is configured to perform morphological opening operation on the mask graph to obtain a connected component graph;

[0036] The calculation subunit is configured to calculate the circular arc degree of the largest connected component region in the connected component graph according to the prior that the pixel density of the pupil region is higher than that of other regions, and then calculate the circular arc degree of the largest connected component region according to the prior that the pupil is circular or elliptical;

[0037] The judgment subunit is configured to, based on the circular arc degree of the largest connected component region, if the largest connected component region is approximately circular or elliptical, it indicates that the largest connected component region is the region where the pupil is located, and take the center of the largest connected component region as the center of the pupil;

[0038] The creation subunit is configured to create a rectangle with the pupil center as the center of the rectangle.

[0039] Preferably, creating a rectangle with the pupil center as the center of the rectangle includes:

[0040] Calculate the longest outer diameter of the largest connected component region according to the value range of the pupil outer diameter, the value range of the iris outer diameter, and the value range of the sclera outer diameter;

[0041] Obtain a rectangle as the cropping region based on 10 times the horizontal direction and 6 times the vertical direction of the longest outer diameter;

[0042] If the rectangle boundary exceeds the image boundary, translate the cropping region inward until the cropping border coincides with the image boundary so that the cropping region is within the image range.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method according to any one of the above.

[0044] The present invention also provides a computer-readable storage medium storing a computer program, which implements the method according to any one of the above when the computer program is executed.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] The present invention provides a method for accurately predicting the fixation point in real time, and by eliminating 80% of irrelevant pixels in the input image and the method of predicting values through multiple levels of neural network output, the computational complexity is reduced in multiple dimensions, thereby reducing the system latency and improving the rendering quality.

[0047] The present invention adaptively adjusts the prediction speed to meet different requirements for real-time performance and accuracy, and the fixation point prediction is accurate. Description of the Drawings

[0048] To more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.

[0049] Figure 1 It is a schematic diagram of eye movement tracking in an embodiment of the present invention;

[0050] Figure 2 It is a schematic flowchart of a fixation point prediction method for a large-space immersive VR tour in an embodiment of the present invention;

[0051] Figure 3 It is a schematic structural diagram of an electronic device in an embodiment of the present invention. Description of the Drawings:

[0053] 1010. Processor; 1020. Memory; 1030. Input / Output Interface; 1040. Communication Interface; 1050. Bus. Specific Embodiments

[0054] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0055] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0056] First, some technical terms used in the present invention will be elaborated:

[0057] 1. Eye movement tracking (fixation point prediction) error

[0058] (1) When the fixation point is located at the center of the view, the radius (in pixels) r of the foveal region f The formula is (as Figure 1 shown):

[0059] r f = r i + e = ρd·tan(θ i + δθ)

[0060] Among them, ρ represents the pixel density of the display, d represents the distance between the fixation point and the screen, and θ iRepresents the eccentric angle of the fovea, θ f = θ i + δθ represents the eccentric angle of the fovea considering the fixation point prediction error δθ. The radius of the foveal region without tracking error is r i = ρd·tan(θ i ), and the error constant e = ρd·(tan(θ f ) = tan(θ i )) is used to represent the change in the radius of the foveal region.

[0061] (2) If the fixation point is not at the center of the view, its actual radius is less than r f , and it can still be rendered according to r f during rendering, and its error is much smaller than the fixation point prediction error δθ, so this error can be ignored.

[0062] (3) Usually, it can be set to θ i = 5.2°, and the inner foveal region, i.e., inter-foveal, is set to 15°. This is equivalent to dividing the screen into three regions by two concentric circles: the inner circle is the foveal region, which is drawn with the highest resolution; the outer circle is the inner foveal region, which is drawn with 1 / 4 resolution; and the other regions outside the outer circle and within the screen are drawn with 1 / 16 resolution.

[0063] 2. Fixation Point Rendering Pipeline

[0064] (1) The fixation point rendering pipeline of modern head-mounted devices generally includes 3 main components: a near-eye camera, an interconnect link (used to connect the near-eye camera and the host processor, such as MIPI C-PHY, which is a physical layer standard developed by the MIPI Alliance for high-speed serial data transmission), and a host processor.

[0065] (2) Fixation Point Rendering Pipeline:

[0066] 1) Use a near-eye monochrome camera to capture an eye image

[0067] 2) Preprocess and read out the image through an Image Signal Processor (ISP)

[0068] 3) Then connect through MIPI C-PHY and transfer the image to the host processor

[0069] 4) After receiving the image, the host processor first sends it to the fixation point prediction network

[0070] 5) The fixation point prediction network estimates the line-of-sight direction

[0071] 6) Use the estimated line-of-sight direction to guide the host processor in the foveal rendering process.

[0072] 3. System Delay

[0073] (1) It takes about 1ms for the camera to sense and capture, and the MIPI C-PHY communication is less than 1ms. The existing fixation prediction network (gaze direction detection) takes about 2ms to 20ms, and the rendering and display are about 2ms or more.

[0074] (2) Expressed by the formula: t total = t sensing + t comm + t tracking + t fr , where t sensing represents the camera sensing delay, t comm represents the communication delay of MIPI C-PHY, t tracking represents the fixation prediction delay, and t fr represents the fixation rendering delay. This method optimizes the design of the DNN network for fixation prediction, thereby reducing t tracking .

[0075] (3) The greater the delay, the lower the frame rate. Generally, the head-mounted display requires a frame rate of at least 60FPS. If the frame rate is too low, it will cause stuttering and dizziness.

[0076] According to the approximate distribution of the fixation rendering delay, the key points of optimization in this method are the fixation prediction network and the rendering process. Controlling the system delay below 16.67ms can achieve real-time rendering at 60FPS.

[0077] Embodiment 1

[0078] As Figure 2 shown, the embodiment of the present invention provides a fixation prediction method for VR large-space immersive tour, and the method includes:

[0079] Obtain the eye image transmitted from the near-eye camera;

[0080] Construct a fixation prediction network model;

[0081] Input the eye image into the fixation prediction network model and output the predicted gaze direction.

[0082] In this embodiment, when the "fixation prediction network" receives the eye image transmitted from the near-eye camera, it first aligns and crops, and then passes the cropped image to the "convolution layer" for shallow feature extraction. After processing, it is passed backward to the "first attention module", "second attention module", "third attention module", and "fourth attention module" for deep feature extraction, and outputs the predicted gaze direction, so as to meet the two requirements of real-time and high quality.

[0083] In this embodiment, the input image is the eye image transmitted by the near-eye camera. The output is the predicted line of sight direction (a two-dimensional value representing a direction vector).

[0084] In this embodiment, (1) the purpose of cropping: Since the eye image collected by the near-eye camera contains redundant pixels that are irrelevant to the fixation point prediction task, such as the background, facial muscles, etc., these pixels have a negative impact on the prediction result and will increase the computational cost. Especially on the head-mounted display device, the increase in computational overhead will affect the latency and frame rate. Therefore, the redundant pixels need to be removed first.

[0085] (2) Cropping algorithm:

[0086] 1) Algorithm principle: First, find the pupil, and then use the pupil as the center of the rectangle to crop most of the irrelevant pixels. Because the pixels around the pupil are the most effective input for the fixation point prediction task.

[0087] 2) First, utilize the prior that the pupil is usually darker than the surrounding sclera and iris, set a threshold, and perform an inverse binary mask operation on the input image (binary means that the pixel is either 0 or 1, and inverse binary means that the black in the original image becomes white, aiming to emphasize the darker areas of the image) to obtain a mask image.

[0088] 3) Perform morphological opening on the above mask image (morphological opening is a combined process of first performing an erosion operation and then a dilation operation. The erosion operation can remove small objects or details in the image, while the dilation operation can restore the image structure eroded to a certain extent. Through the combination of these two operations, the opening operation can achieve effects such as removing noise, separating connected objects, and smoothing the boundaries of larger objects) to reduce the noise in the image, so that only the pixels in darker areas such as the pupil and eyelashes will be retained. Through morphological opening, a connected components map is obtained.

[0089] 4) According to the prior that the pixel density in the pupil area is higher than that in other areas, by searching for the largest connected component area in the connected components map, and then according to the prior that the pupil is circular or elliptical, calculate the circular arc degree of this largest connected component area. The formula is:

[0090]

[0091] where A represents the area of the connected component, P represents the boundary perimeter of the connected component. When C is close to 1, it indicates that this area is close to a circle. A threshold can be set When When it is, it indicates that this connected component is circular or elliptical (let ). The area and perimeter of the connected component can be calculated using the regionprops function in MATLAB software. Other algorithms can also be used.

[0092] 5) If the connected component is approximately circular or elliptical, it indicates that this connected component region is the pupil region. The center of this region can be used as the center of the pupil. So far, the position of the pupil center has been found through the algorithm. If the largest connected component region is not approximately circular or elliptical, it means the image quality is not high, and it is directly discarded, that is, no further processing is performed on this input image, and the next image can be processed.

[0093] 6) Finally, create a rectangle with the pupil center as the center of the rectangle

[0094] According to prior knowledge: the value range of the outer diameter of the pupil is 2.5 - 4 mm, the value range of the outer diameter of the iris is about 11.5 mm, and the value range of the outer diameter of the sclera is about 23.5 mm. Therefore, first calculate the longest outer diameter of the above-mentioned largest connected component, and then obtain a rectangle as the cropping region according to 10 times the horizontal and 6 times the vertical of this length.

[0095] If the boundary of this rectangle exceeds the image boundary, the cropping region is translated inward until the cropping border coincides with the image boundary to ensure that the cropping region is within the image range.

[0096] Generally speaking, the width of the cropping region can be reduced to 1 / 4 of the original, the height can be reduced to 1 / 5 of the original, and the total area can be reduced to 1 / 20 of the original, with an effective reduction range reaching 95%. This greatly reduces the image size input to the subsequent network, which is beneficial to significantly reducing latency, improving computing performance, and meeting the requirements of real-time computing.

[0097] In this embodiment, there are a total of 5 convolutional layers in the architecture diagram, but 3 attention modules are not drawn, and each module has been iterated 2 times. So if all are counted, there are a total of 12.

[0098] (1) In this method, the structures of all convolutional layers are the same, all composed of a residual network, and their functions are also the same, which is downsampling, reducing the size of the feature map by half and increasing the number of channels by 4 times. Because exists, after the first iteration, the number of channels is 2 times the original; after the second iteration, the number of channels is 4 times the original.

[0099] (2) I con,1 In addition to downsampling, it also has the function of shallow feature extraction.

[0100] (3) The input image What the first convolutional layer outputs is

[0101] Among them, H and W are respectively the height and width of the size of the input image, and C is the number of channels. Generally, C = 3, that is, the three RGB channels.

[0102] (4) The situation of the second convolutional layer is slightly more complicated. The second convolutional layer is inside the "first attention module". Since the "first attention module" iterates twice, during the first iteration, During the second iteration,

[0103] (5) The third convolutional layer The fourth convolutional layer The fifth convolutional layer

[0104] In this embodiment, (1) The linear layer is also called the fully connected layer, and its main function is to perform a linear transformation on the input data. The line-of-sight direction is directly output from the linear layer.

[0105] (2) Each "attention module" has a linear layer, so each will output a predicted line-of-sight direction; and because each "attention module" iterates 2 times, each "attention module" finally outputs a total of two predicted line-of-sight directions.

[0106] (3) Not every layer of the attention module predicts the line-of-sight direction. Only one is output from the "first attention module" to meet the real-time requirement; and then another one is output for a more accurate prediction of the line-of-sight method.

[0107] (4) Specifically, a time threshold ε is set. When the final "second line-of-sight direction" is not generated within this time threshold (possibly due to reasons such as a complex current scene or network latency), the "first line-of-sight direction" is directly used. It can also be set according to user preferences. For example, if the user requires low latency, the time threshold is reduced; if the user requires high quality, the time threshold is increased, or even the "second line-of-sight direction" is directly used.

[0108] (5) There is also another situation. If the "second line-of-sight direction" is generated before the next prediction, it is necessary to determine whether the "first line-of-sight direction" and the "second line-of-sight direction" are within the foveal region. If they are, the processing is still carried out according to the "first line-of-sight direction"; if not, it means the deviation is relatively large. At this time, it is necessary to redraw according to the "second line-of-sight direction". If the next prediction result comes out before the completion of the redrawing, the line-of-sight direction of the next prediction is used for drawing, ignoring the current redrawing.

[0109] (6) Additionally, a skip connection is added to the architecture diagram, connecting the output of the "First Attention Module" to the input of the "Fourth Attention Module", with a line drawn in between. The purpose is similar to that of U-Net, which is to fuse the features extracted from the shallow layer with those from the deep layer, so that the prediction result can consider both local and global information simultaneously, thereby obtaining a more accurate prediction result.

[0110] In this embodiment, (1) the "First Attention Module", "Second Attention Module", "Third Attention Module", and "Fourth Attention Module" have exactly the same structure. The reason for adding 4 of them and each of them performing an iteration on its own is mainly to better learn deep image features. Moreover, by inserting a convolutional layer, the size of the feature map is reduced and the number of channels is increased, thereby effectively increasing the receptive field of the convolutional kernel and enabling learning of more global information.

[0111] (3) The most core part of the "First Attention Module" is the "Hybrid Attention Module", which is basically similar to the standard Transformer but has two differences.

[0112] 1) The first difference is that at the position of the multi-head attention module in the standard Transformer, we use a channel attention module in parallel with it (the specific implementation method of this channel attention module is not limited, such as the Squeeze-and-Excitation (SE) module, Efficient Channel Attention (ECA) module, and their variants, etc.).

[0113] 2) The second difference is that we use a mechanism of moving three small windows of different sizes on the feature map (in principle, borrowing the window moving mechanism of Swin Transformer). The sizes of these three small windows are and And padding is used to fill the surrounding of the image (padding is only done when the image size is not a multiple of the small window size). Then, cross-attention is calculated inside the small windows. And to solve the problem of pixel connectivity between adjacent small windows, we adopt a method of gradually increasing the size of the three small windows by a factor of two, thereby increasing the receptive field and enabling pixels originally in two small windows to be inside a larger window, so as to calculate cross-attention.

[0114] (4) The symbol represents feature map fusion, and the fusion method is that the size remains unchanged while the number of channels doubles.

[0115] (5) The "First Attention Module" is iterated 2 times. During the second iteration, the symbol The output feature map is used as the input and fed into the "hybrid attention module". In the skip connection, it serves as one of the inputs for the next feature fusion. Similarly, the other attention modules are also iterated twice respectively.

[0116] In this embodiment, (1) the average fixation point prediction error

[0117]

[0118] where B represents the set of batches of training data, represents the training data set of batch b, y d and respectively represent the predicted and true line-of-sight directions of training data d.

[0119] (2) To address the problem of a relatively large 95th percentile of the fixation point error, we adopt minimizing the maximum tracking error during the training process to solve this problem

[0120]

[0121] where B represents the set of batches of the training data set, represents the training data set in batch b, θ d and respectively represent the predicted line-of-sight direction (in radians) and the true line-of-sight direction (in radians) of the input sample d in the training data set. N is a scaling factor for controlling the approximation. During the training process, the value of N is carefully adjusted to adapt to the value distribution of the input training data, so as to ensure better convergence of the training process.

[0122] (3) The total loss function

[0123] (4) To obtain better training results, we adopt different training strategies:

[0124] Train the sub-network of "the first line-of-sight direction" with to make the first line-of-sight direction trained faster and meet the accuracy of 95%, so as to quickly predict the result.

[0125] For the entire network, that is, "the second line-of-sight direction", use to train, with the corresponding λ = 0.8, to make play a greater role, so as to more accurately predict the result and solve the problem of accuracy above 95%.

[0126] Embodiment 2

[0127] The present invention also provides a fixation point prediction system for VR large-space immersive tours. The system is used to implement the method described in any one of the above-mentioned Embodiment 1. The system includes: an acquisition module, a construction module, and a prediction module;

[0128] The acquisition module is used to acquire the eye image transmitted by the near-eye camera;

[0129] The construction module is used to construct a fixation point prediction network model;

[0130] The prediction module is used to input the eye image into the fixation point prediction network model and output the predicted line of sight direction.

[0131] In this embodiment, the prediction module includes: a cropping unit, a first extraction unit, and a second extraction unit;

[0132] The cropping unit is used to align and crop the eye image;

[0133] The first extraction unit is used to transfer the cropped image to a convolutional layer for shallow feature extraction;

[0134] The second extraction unit is used to process and then transfer it backward to the first attention module, the second attention module, the third attention module, and the fourth attention module for deep feature extraction, and output the predicted line of sight direction.

[0135] In this embodiment, the cropping unit includes: a mask sub-unit, an operation sub-unit, a calculation sub-unit, a judgment sub-unit, and a creation sub-unit;

[0136] The mask sub-unit is used to perform an inverse binary mask operation on the eye image based on prior knowledge and a preset threshold to obtain a mask image;

[0137] The operation sub-unit is used to perform a morphological opening operation on the mask image to obtain a connected component image;

[0138] The calculation sub-unit is used to calculate the circular arc degree of the largest connected component region by searching for the largest connected component region in the connected component image according to the prior that the pixel density of the pupil region is higher than other regions, and then according to the prior that the pupil is circular or elliptical;

[0139] The judgment sub-unit is used to, based on the circular arc degree of the largest connected component region, if the largest connected component region is approximately circular or elliptical, it indicates that the largest connected component region is the region where the pupil is located, and take the center of the largest connected component region as the center of the pupil;

[0140] The creation sub-unit is used to create a rectangle with the pupil center as the center of the rectangle.

[0141] In this embodiment, with the pupil center as the center of the rectangle, a rectangle is created, including:

[0142] According to the value range of the pupil outer diameter, the value range of the iris outer diameter, and the value range of the sclera outer diameter, calculate the longest outer diameter of the largest connected component region;

[0143] Based on 10 times the horizontal direction and 6 times the vertical direction of the longest outer diameter, obtain a rectangle as the cropping region;

[0144] If the rectangle boundary exceeds the image boundary, translate the cropping region inward until the cropping border coincides with the image boundary, so that the cropping region is within the image range.

[0145] Embodiment Three

[0146] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method for restoring the three-dimensional structure of a high-quality urban renewal landscape building described in any one of the above embodiments.

[0147] Figure 3 Figure 1 shows a more specific schematic diagram of the hardware structure of the electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. Among them, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other inside the device through the bus 1050.

[0148] The processor 1010 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0149] The memory 1020 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0150] The input / output interface 1030 is used to connect to the input / output module to achieve information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Among them, the input device can include a keyboard, mouse, touch screen, microphone, various sensors, etc., and the output device can include a display, speaker, vibrator, indicator light, etc.

[0151] The communication interface 1040 is used to connect to the communication module (not shown in the figure) to achieve communication interaction between this device and other devices. Among them, the communication module can achieve communication through wired means (such as USB (Universal Serial Bus), network cable, etc.) or through wireless means (such as mobile network, WIFI (Wireless Fidelity), Bluetooth, etc.).

[0152] The bus 1050 includes a path for transmitting information between various components of the device (such as the processor 1010, memory 1020, input / output interface 1030, and communication interface 1040).

[0153] It should be noted that although the above device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solution of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0154] The system of the above embodiment is used to implement the three-dimensional structure restoration method of a corresponding high-quality urban renewal landscape building in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0155] Embodiment 4

[0156] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present disclosure also provides a non-transitory computer-readable storage medium, and the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to make the computer execute a three-dimensional structure restoration method of a high-quality urban renewal landscape building as described in any of the foregoing embodiments.

[0157] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0158] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute a three-dimensional structure restoration method of a high-quality urban renewal landscape building as described in any one of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0159] Those of ordinary skill in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples; under the concept of the present disclosure, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present disclosure as described above, and they are not provided in detail for the sake of brevity.

[0160] In addition, for the sake of simplicity of description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. In addition, the devices may be shown in block diagram form in order to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (i.e., these details should be fully within the understanding of those skilled in the art). In the case where specific details (such as circuits) are set forth to describe the exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with variations of these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0161] Although the present disclosure has been described in connection with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0162] Thus, the units of the examples described in the embodiments of the present application can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0163] The embodiments described above are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solution of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for predicting fixation points in a VR large-space immersive tour, characterized in that, The method includes: Obtaining an eye image transmitted by a near-eye camera; Constructing a fixation point prediction network model; Inputting the eye image into the fixation point prediction network model and outputting a predicted line of sight direction.

2. The method according to claim 1, wherein Inputting the eye image into the fixation point prediction network model and outputting a predicted line of sight direction includes: Aligning and cropping the eye image; Transmitting the cropped image to a convolutional layer for shallow feature extraction; After processing, transmitting it backward to the first attention module, the second attention module, the third attention module, and the fourth attention module for deep feature extraction and outputting a predicted line of sight direction.

3. The method according to claim 2, characterized in that, Cropping the aligned eye image includes: Performing an inverse binary mask operation on the eye image based on prior knowledge and a preset threshold to obtain a mask image; Performing a morphological opening operation on the mask image to obtain a connected component image; According to the prior that the pixel density of the pupil area is higher than that of other areas, by searching for the largest connected component area in the connected component image, and then according to the prior that the pupil is circular or elliptical, calculating the circular arc degree of the largest connected component area; Based on the circular arc degree of the largest connected component area, if the largest connected component area is approximately circular or elliptical, it indicates that the largest connected component area is the area where the pupil is located, and taking the center of the largest connected component area as the center of the pupil; Creating a rectangle with the pupil center as the rectangle center.

4. The method according to claim 3, wherein Creating a rectangle with the pupil center as the rectangle center includes: Calculating the longest outer diameter of the largest connected component area according to the value range of the pupil outer diameter, the value range of the iris outer diameter, and the value range of the sclera outer diameter; Obtaining a rectangle as the cropping area based on 10 times the horizontal and 6 times the vertical of the longest outer diameter; If the rectangle boundary exceeds the image boundary, translating the cropping area inward until the cropping border coincides with the image boundary to make the cropping area within the image range.

5. A fixation point prediction system for VR large-space immersive tours, the system being used to implement the method described in any one of claims 1-4, characterized in that, The system includes: an acquisition module, a construction module, and a prediction module; The acquisition module is used to obtain an eye image transmitted by a near-eye camera; The construction module is used to construct a fixation point prediction network model; The prediction module is used to input the eye image into the fixation point prediction network model and output a predicted line of sight direction.

6. The system according to claim 5, wherein The prediction module includes: a cropping unit, a first extraction unit, and a second extraction unit; The cropping unit is used to align and crop the eye image; The first extraction unit is used to transmit the cropped image to a convolutional layer for shallow feature extraction; The second extraction unit is used to transmit it backward to the first attention module, the second attention module, the third attention module, and the fourth attention module for deep feature extraction and output a predicted line of sight direction.

7. The system according to claim 6, characterized in that, The cropping unit includes: a mask subunit, an operation subunit, a calculation subunit, a judgment subunit, and a creation subunit; The mask subunit is used to perform an inverse binary mask operation on the eye image based on prior knowledge and a preset threshold to obtain a mask image; The operation subunit is used to perform a morphological opening operation on the mask image to obtain a connected component image; The calculation subunit is configured to calculate the circular arc degree of the largest connected region in the connected component graph according to the prior that the pixel density of the pupil region is higher than that of other regions, and then calculate the circular arc degree of the largest connected region according to the prior that the pupil is circular or elliptical; The judgment subunit is configured to, based on the circular arc degree of the largest connected region, if the largest connected region is approximately circular or elliptical, it indicates that the largest connected region is the region where the pupil is located, and use the center of the largest connected region as the center of the pupil; The creation subunit is configured to create a rectangle with the pupil center as the center of the rectangle.

8. The system according to claim 7, wherein Creating a rectangle with the pupil center as the center of the rectangle includes: Calculating the longest outer diameter of the largest connected region according to the value range of the pupil outer diameter, the value range of the iris outer diameter, and the value range of the sclera outer diameter; Obtaining a rectangle as the cropping region based on 10 times the horizontal direction and 6 times the vertical direction of the longest outer diameter; If the rectangle boundary exceeds the image boundary, the cropping region is translated inward to the image until the cropping border coincides with the image boundary, so that the cropping region is within the image range.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method described in any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed, the method described in any one of claims 1 to 4 is implemented.

Images