Focus depth determination method and related device

Abstract

The invention discloses a focus depth determination method and a related device, and relates to the technical field of rendering, and the method comprises the steps: collecting eye movement data of a user according to a preset eye movement sampling rate in a process that the user watches a three-dimensional scene, so as to obtain multi-frame eye movement data collected in a target duration, and determining a focus depth of the user according to the multi-frame eye movement data, determining a fixation point position of the sight line of the user in the target three-dimensional model, generating a sphere with the radius as a target value by taking the fixation point position as a sphere center, obtaining a target vertex set in the sphere in the target three-dimensional model, and performing vertex density analysis on the target vertex set to obtain a focus depth for rendering the three-dimensional scene. According to the method, the focus depth dynamically adjusted along with the sight line of the user can be generated, the detail resolution capability of the user on the fixation point is improved, and the watching experience of the user on the three-dimensional scene is further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of rendering, in particular to a focal depth determination method and related device. BACKGROUND

[0002] The three-dimensional models in the current virtual reality (VR) scene (such as VR game scene, guided VR tour scene, etc.) are usually rendered in an offline rendering manner. However, discrete rendering is a rendering manner with fixed focus, which leads to fixed blur areas and non-blur areas in the rendered three-dimensional models, and cannot dynamically adjust according to the user's line of sight, thereby reducing the user's viewing experience of the three-dimensional models. SUMMARY

[0003] In view of the above problems, the present application provides a focal depth determination method and related device to achieve the purpose of generating focal depth dynamically adjusted according to the user's line of sight. The specific scheme is as follows:

[0004] The first aspect of the present application provides a focal depth determination method, comprising:

[0005] In the process of the user watching a three-dimensional scene, the eye movement data of the user is collected according to a preset first eye movement sampling rate to obtain a plurality of frames of eye movement data collected within a target time length;

[0006] According to the plurality of frames of eye movement data, the gaze point position of the user's line of sight in a target three-dimensional model is determined, wherein the target three-dimensional model refers to a three-dimensional model gazed at by the user in the three-dimensional scene, and the target three-dimensional model includes a plurality of vertices;

[0007] A sphere with a target value as a radius is generated with the gaze point position as a sphere center, and a vertex set in the target three-dimensional model located in the sphere is obtained as a target vertex set, wherein the target vertex set includes position data of each of the plurality of vertices;

[0008] The target vertex set is subjected to vertex density analysis to obtain a focal depth for rendering the three-dimensional scene.

[0009] In a possible implementation, the determining of the gaze point position of the user's line of sight in the target three-dimensional model according to the plurality of frames of eye movement data comprises:

[0010] The plurality of frames of eye movement data are evenly divided into a plurality of groups in the order of collection time, and the gaze object corresponding to each group is determined according to the eye movement data in each group to obtain the gaze objects corresponding to the plurality of groups respectively;

[0011] determine whether there is a target gaze object whose gaze ratio reaches a preset ratio threshold, wherein the gaze ratio of a gaze object is a ratio of a gaze number of the gaze object to a total gaze number;

[0012] If yes, calculate a distribution standard deviation of gaze points of the user according to the multiple frames of eye movement data;

[0013] If the distribution standard deviation is less than a preset standard deviation threshold, take a three-dimensional model of the target gaze object as the target three-dimensional model, and obtain model data of the target three-dimensional model;

[0014] Determine the gaze point position according to the multiple frames of eye movement data and the model data.

[0015] In a possible implementation, the determining of the gaze point position according to the multiple frames of eye movement data and the model data includes:

[0016] Process each frame of eye movement data in the multiple frames of eye movement data into a ray equation of a user's line of sight to obtain multiple ray equations;

[0017] Construct a hierarchical bounding box tree according to the model data, wherein a plurality of vertices of the target three-dimensional model are connected to form a plurality of polygons, and each leaf node of the hierarchical bounding box tree represents one of the polygons;

[0018] For each ray equation in the multiple ray equations: perform collision detection according to the ray equation and the hierarchical bounding box tree to obtain a leaf node of the hierarchical bounding box tree that intersects with the ray equation, and take a polygon represented by the leaf node as a polygon corresponding to the ray equation; and obtain the polygons respectively corresponding to the multiple ray equations;

[0019] Determine, from the polygons respectively corresponding to the multiple ray equations, a polygon that is closest to a distance of a device for collecting the eye movement data as a target polygon;

[0020] Determine, according to the model data, coordinate data of each vertex of the target polygon;

[0021] Determine, according to the coordinate data of each vertex of the target polygon, coordinate data of a center point of the target polygon as the gaze point position, wherein the center point is equidistant from the vertices.

[0022] In a possible implementation, the method further includes:

[0023] In a case where the distribution standard deviation is less than the standard deviation threshold, the collecting of the eye movement data of the user at the preset first eye movement sampling rate to obtain a plurality of frames of eye movement data collected within a target time length is returned while the three-dimensional model of the target fixation object is taken as the target three-dimensional model and the model data of the target three-dimensional model is acquired.

[0024] In a case where the distribution standard deviation is greater than or equal to the standard deviation threshold or the target fixation object does not exist, the three-dimensional model of the target fixation object is not taken as the target three-dimensional model, the model data of the target three-dimensional model is not acquired, and the collecting of the eye movement data of the user is continued at a second eye movement sampling rate, where the second eye movement sampling rate is less than the first eye movement sampling rate.

[0025] In a possible implementation, the vertex density analysis on the target vertex set to obtain a focal depth for rendering the three-dimensional scene includes:

[0026] Performing vertex density clustering processing on the target vertex set to obtain a target vertex cluster with the largest number of vertices;

[0027] Performing vertex density weighting processing on the target vertex cluster to obtain the focal depth.

[0028] In a possible implementation, the vertex density clustering processing on the target vertex set to obtain a target vertex cluster with the largest number of vertices includes:

[0029] Determining the number of vertices in the target vertex set as the number of vertices inside a sphere;

[0030] Determining the density of the sphere according to the number of vertices inside the sphere;

[0031] Obtaining a clustering radius according to the density of the sphere, where the clustering radius and the density of the sphere are inversely related;

[0032] Performing clustering on the target vertex set based on the clustering radius to obtain a plurality of vertex clusters;

[0033] Determining a vertex cluster with the largest number of vertices from the plurality of vertex clusters as the target vertex cluster.

[0034] In a possible implementation, the vertex density weighting processing on the target vertex cluster to obtain the focal depth includes:

[0035] Generating a minimum bounding box of the target vertex cluster and determining the volume of the minimum bounding box;

[0036] Determining the number of vertices contained in the target vertex cluster;

[0037] determine a local density according to a number of vertices contained in the target vertex cluster and a volume of the minimum bounding box;

[0038] obtain a depth value of each vertex in the target vertex cluster from position data of each vertex in the target vertex cluster;

[0039] perform a weighted average according to the local density and the depth value of each vertex in the target vertex cluster to obtain the focal depth.

[0040] The second aspect of the present application provides a focal depth determination device, comprising:

[0041] an eye movement data acquisition module configured to collect eye movement data of a user at a preset first eye movement sampling rate during the user's viewing of a three-dimensional scene to obtain a plurality of frames of eye movement data collected within a target time period;

[0042] a gaze point position determination module configured to determine a gaze point position of a user's line of sight in a target three-dimensional model according to the plurality of frames of eye movement data, wherein the target three-dimensional model refers to a three-dimensional model on which the user gazes in the three-dimensional scene, and the target three-dimensional model includes a plurality of vertices;

[0043] a sphere-in-vertex determination module configured to generate a sphere with a radius of a target value with the gaze point position as a sphere center, obtain a vertex set in the target three-dimensional model that is located in the sphere as a target vertex set, and wherein the target vertex set includes position data of each vertex in the target vertex set;

[0044] a focal depth determination module configured to perform vertex density analysis on the target vertex set to obtain a focal depth for rendering the three-dimensional scene.

[0045] The third aspect of the present application provides a computer program product, comprising computer readable instructions, when the computer readable instructions are run on an electronic device, the electronic device implements the focal depth determination method of the first aspect or any implementation manner of the first aspect.

[0046] The fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0047] the memory is configured to store a computer program;

[0048] the processor is configured to execute the computer program to enable the electronic device to implement the focal depth determination method of the first aspect or any implementation manner of the first aspect.

[0049] The fifth aspect of the present application provides a computer storage medium, the storage medium carries one or more computer programs, when the one or more computer programs are executed by an electronic device, the electronic device can execute the focus depth determination method of the first aspect or any implementation manner of the first aspect.

[0050] By the above technical solution, the focus depth determination method provided by the present application collects the eye movement data of the user according to the preset eye movement sampling rate in the process of the user watching the three-dimensional scene, to obtain a plurality of frames of eye movement data collected in a target time length, and then determines the gaze point position of the user's line of sight in the target three-dimensional model according to the plurality of frames of eye movement data. Considering that the vertex density around the gaze point can represent the model complexity of the gaze point area, and the more complex the model is, the more it needs to reduce the focus depth to achieve the purpose of keeping the aligned plane clear and reducing the clarity of the non-aligned plane, thereby avoiding the visual interference of the non-aligned plane and improving the user's detail resolution capability at the gaze point. Based on this, the present application can generate a sphere with the gaze point position as the center and the target value of the radius, obtain a target vertex set in the sphere in the target three-dimensional model, perform vertex density analysis on the target vertex set, and obtain the focus depth for rendering the three-dimensional scene. As can be seen, the present application can generate the focus depth dynamically adjusted according to the user's line of sight, improve the user's detail resolution capability at the gaze point, and further improve the user's viewing experience of the three-dimensional scene. BRIEF DESCRIPTION OF DRAWINGS

[0051] The above and other features, advantages, and aspects of the present disclosure will become more apparent by describing in detail the embodiments thereof with reference to the attached drawings. Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. It should be understood that the drawings are schematic and elements and features are not necessarily to scale.

[0052] Figure 1 A flowchart of a focus depth determination method provided by the present application;

[0053] Figure 2 A structural diagram of a focus depth determination device provided by the present application;

[0054] Figure 3 A structural diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0055] The embodiments of the present application are described below with reference to the accompanying drawings. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0056] The embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art can know that, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0057] The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, and this is only a way of distinguishing the objects with the same attributes in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or equipment containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or equipment.

[0058] In order for those skilled in the art to better understand the present application, the related terms applied in the present application are explained as follows.

[0059] Depth of focus: abbreviated as DOF, refers to the distance that the focus is allowed to move along the optical axis while maintaining the clarity of the image. The larger the depth of focus, the wider the tolerance range of defocus, but the resolution may be reduced; the smaller the depth of focus, only the area near the focusing plane is clear, and the detail resolution is higher.

[0060] In three-dimensional rendering, the depth of focus is a key parameter for controlling the depth of field effect. The depth of field effect is achieved by simulating the depth of focus in an optical system, so that specific objects or regions in the scene remain clear, while other objects or regions appear blurred, thereby enhancing the level of detail and realism of the image. For example, in 3ds Max (a professional three-dimensional modeling, animation, rendering and special effect production software), by adjusting the depth of focus and depth of field parameters of the camera, it can be accurately controlled which objects are in clear state and which objects are in blurred state, so that the rendered image has more artistic expression.

[0061] The present application provides a depth of focus determination method, which can be applied to a terminal or a server with data processing capability, and can also be applied to a system composed of the above-mentioned terminal and server.

[0062] Of course, the depth of focus determination method provided by the present application can also be applied to other, which is not limited by the present application.

[0063] In order for those skilled in the art to better understand the present application, the depth of focus determination method of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0064] Reference Figure 1 , Figure 1A flowchart of a focal depth determination method provided by an embodiment of the present application is shown in FIG. 1. The focal depth determination method can include the following steps. Figure 1

[0065] In step S101, during the user's viewing of the three-dimensional scene, the eye movement data of the user is collected at a preset first eye movement sampling rate to obtain a plurality of frames of eye movement data collected within a target time length.

[0066] In this embodiment, the user can wear an eye movement tracking device during the viewing of the three-dimensional scene, and the first eye movement sampling rate can be set so that the eye movement tracking device can collect the eye movement data of the user at the preset first eye movement sampling rate.

[0067] Considering that the error of a single frame of eye movement data is large, for example, the user often scans the scene during the viewing of the three-dimensional scene, and it is difficult to identify the user's scanning of the scene from a single frame of eye movement data, resulting in poor accuracy of the focal depth when using a single frame of data to identify the focal depth. Based on this, the eye movement tracking device can be used to collect a plurality of frames of eye movement data within a target time length to more accurately identify the focal depth.

[0068] Optionally, a ring-shaped data buffer can be established to dynamically save a plurality of frames of eye movement data within a target time length, for example, with a first eye movement sampling rate of 120 Hz (Hertz) and a target time length of 2 seconds, 240 frames of eye movement data can be stored in the ring-shaped data buffer.

[0069] Of course, the above-mentioned first eye movement sampling rate and target time length can be other values, which can be set according to the actual scene, and are not limited here.

[0070] Optionally, the above-mentioned eye movement tracking device can be a head-mounted eye movement tracking device, such as Varjo XR-4, HTC Vive Pro Eye, Meta Quest Pro, etc.; optionally, the eye movement tracking device can also be an external eye movement tracking device, such as Tobii Eye Tracker 5, etc.

[0071] In step S102, the gaze point position of the user's line of sight in the target three-dimensional model is determined according to the plurality of frames of eye movement data.

[0072] It can be understood that each frame of eye movement data collected by the eye movement tracking device includes gaze point coordinates, confidence, and a timestamp. In order to eliminate device differences, unify data scales, simplify calculation logic, and provide a standardized basis for subsequent analysis, it is usually necessary to convert the gaze point coordinates into normalized screen coordinates. Specifically, the gaze point coordinates are normalized to obtain normalized screen coordinates. It is worth noting that the normalized screen coordinates are two-dimensional coordinates. ​

[0073] Therefore, the embodiment can determine the gaze point position of the user's line of sight in the target three-dimensional model according to the multi-frame normalized screen coordinates, that is, the embodiment can fuse the user's line of sight in the target time length into a gaze point and obtain the position data of the gaze point.

[0074] Here, the three-dimensional model in which the gaze point is located is defined as the target three-dimensional model, that is, the target three-dimensional model refers to the three-dimensional model gazed at by the user in the three-dimensional scene.

[0075] It can be understood that in the three-dimensional scene, each object object corresponds to a three-dimensional model, and each three-dimensional model includes a plurality of vertices which can form a plurality of polygons (such as triangles) by connection. Based on this, the target three-dimensional model includes a plurality of vertices. Here, "a plurality of" refers to a value greater than or equal to 3, and the specific value size is related to the three-dimensional model, which is not limited in the present application.

[0076] Step S103, generating a sphere with the gaze point position as the center and the target value as the radius, obtaining a vertex set in the target three-dimensional model located in the sphere as a target vertex set.

[0077] Among them, the target vertex set includes the position data of each of the plurality of vertices.

[0078] It is considered that the vertex density of the gaze point area is closely related to the depth of focus. The more concentrated the gaze point area is, the more it is necessary to reduce the definition of the peripheral area of the gaze point area to reduce the visual interference of the peripheral area on the gaze point area and improve the user's ability to distinguish details in the gaze point area. The depth of focus can reduce the range of clear imaging (i.e. limit the clear area), so that only the plane strictly aligned with the gaze point maintains high definition, and the misaligned plane is naturally blurred because it is out of the depth of focus. Based on this, the embodiment can analyze the vertex density of the gaze point area to obtain the depth of focus for rendering the target three-dimensional model, that is, the depth of focus for rendering the three-dimensional scene.

[0079] In order to identify the gaze point area, the gaze point position can be used as the center and the preset target value as the radius to generate a sphere, and the area in the sphere can be used as the gaze point area. Therefore, the vertex set in the target three-dimensional model located in the sphere can be obtained, and for the convenience of subsequent introduction, the vertex set is defined as the target vertex set.

[0080] It can be understood that in the model data of the three-dimensional model, the position, normal and other data of each vertex are usually stored in the form of a vertex buffer and an index buffer. Therefore, in this embodiment, the index can be queried from the index buffer, and the position data of all vertices in the target vertex set can be obtained by querying the vertex position data from the vertex buffer through the index.

[0081] For example, the target three-dimensional model includes several triangles, and the first vertex of the ith triangle falls within the above-mentioned sphere. The index Index0 of the first vertex of the ith triangle is Index0 = IndexBuffer[3i], which means that the 3i index is queried from the index buffer of the target triangle to obtain Index0. Based on Index0, the coordinates of the first vertex of the ith triangle can be read from the vertex buffer of the target three-dimensional model. Similarly, the second vertex of the ith triangle falls within the above-mentioned sphere. The index Index1 of the second vertex of the ith triangle is Index1 = IndexBuffer[3i+1], which means that the 3i+1 index is queried from the index buffer of the target triangle to obtain Index1. Based on Index1, the coordinates of the second vertex of the ith triangle can be read from the vertex buffer of the target three-dimensional model. The third vertex of the ith triangle falls within the above-mentioned sphere. The index Index2 of the third vertex of the ith triangle is Index2 = IndexBuffer[3i+2], which means that the 3i+2 index is queried from the index buffer of the target triangle to obtain Index2. Based on Index2, the coordinates of the third vertex of the ith triangle can be read from the vertex buffer of the target three-dimensional model.

[0082] In step S104, the target vertex set is subjected to vertex density analysis to obtain the focal depth for rendering the three-dimensional scene.

[0083] In three-dimensional rendering, the role of the focal depth at least includes: first, highlighting the main body: by adjusting the focal depth, the main object is clear and the background is blurred, thereby guiding the attention of the audience; second, enhancing the sense of hierarchy: by controlling the focal depth, different objects in the three-dimensional scene present different degrees of clarity, thereby enhancing the stereoscopic and spatial sense of the image; third, simulating the real photography effect: by adjusting the focal depth and the depth of field parameters, the focusing and depth of field effect in real photography is simulated, so that the rendered image is more realistic.

[0084] In order to make the area where the user gazes perceive clearer rendering effect, the target vertex set can be subjected to vertex density analysis in this embodiment to obtain the focal depth for rendering the three-dimensional scene.

[0085] Next, the embodiment can render the three-dimensional field based on the focal depth to obtain a rendered frame. Optionally, a hardware synchronization signal can be used to ensure that the eye movement data timestamp is accurately aligned with the rendered frame (specifically, to ensure that the timestamp of the last frame of eye movement data in the plurality of frames of eye movement data is accurately aligned with the rendered frame) to compensate for processing delay.

[0086] The focal depth determination method provided in the application collects eye movement data of a user at a preset eye movement sampling rate during the user's viewing of a three-dimensional scene to obtain a plurality of frames of eye movement data collected within a target time length, and then determines a gaze point position of a user's line of sight in a target three-dimensional model based on the plurality of frames of eye movement data. It is considered that the vertex density around the gaze point can represent the model complexity of the gaze point area, and the more complex the model is, the more it needs to reduce the focal depth to achieve the purpose of keeping the aligned plane clear and reducing the clarity of the non-aligned plane, thereby avoiding visual interference of the non-aligned plane and improving the user's detail resolution capability at the gaze point. Based on this, the application can generate a sphere with a radius of a target value with the gaze point position as the center of the sphere, obtain a target vertex set in the sphere in the target three-dimensional model, perform vertex density analysis on the target vertex set, and obtain the focal depth for rendering the three-dimensional scene. As can be seen, the application can generate a focal depth dynamically adjusted according to the user's line of sight, improve the user's detail resolution capability at the gaze point, and further improve the user's viewing experience of the three-dimensional scene.

[0087] In some embodiments of the application, the process of the foregoing step S102 "determining a gaze point position of a user's line of sight in a target three-dimensional model based on the plurality of frames of eye movement data" is described in detail.

[0088] The user may have a short time to view multiple object objects, in order to ensure the consistency and stability of the user's viewing object, optionally, the embodiment can first determine whether the plurality of frames of eye movement data meet a preset valid gaze determination condition, if so, determine a three-dimensional model that meets the valid gaze determination condition as a target three-dimensional model, and then determine a gaze point position of a user's line of sight in the target three-dimensional model based on the plurality of frames of eye movement data.

[0089] Optionally, the valid gaze determination condition includes one or more of the following conditions: condition 1 is that the total time length of the user's gaze at one three-dimensional model in the three-dimensional scene within the target time length reaches a preset time length threshold, and condition 2 is that the gaze stability of the user's gaze at one three-dimensional model in the three-dimensional scene within the target time length reaches a preset stability threshold.

[0090] Taking the example that the valid gaze determination condition includes the above-mentioned condition 1 and condition 2, the process of "determining a gaze point position of a user's line of sight in a target three-dimensional model based on the plurality of frames of eye movement data" can include:

[0091] The multi-frame eye movement data is divided into multiple groups in time sequence, and the gaze objects corresponding to each group are determined according to the eye movement data in each group, so as to obtain the gaze objects corresponding to the multiple groups respectively, wherein the gaze object refers to the object object gazed by the user in the three-dimensional scene;

[0092] According to the gaze objects corresponding to the multiple groups respectively, it is judged whether there is a target gaze object whose gaze ratio reaches a preset ratio threshold, wherein the gaze ratio of a gaze object is the ratio of the gaze times of the gaze object to the total gaze times;

[0093] If yes, the distribution standard deviation of the user's gaze point is calculated according to the multi-frame eye movement data;

[0094] If the distribution standard deviation is less than a preset standard deviation threshold, the three-dimensional model of the target gaze object is taken as a target three-dimensional model, and the model data of the target three-dimensional model is obtained;

[0095] The gaze point position is determined according to the multi-frame eye movement data and the model data.

[0096] The above-mentioned gaze times and total gaze times are obtained by counting the times of the gaze objects corresponding to the multiple groups respectively, the preset ratio threshold is determined according to the preset time threshold, and the standard deviation threshold is obtained according to the display screen size of the display device of the three-dimensional scene.

[0097] In order to make the person skilled in the art understand the above process better, the following description is exemplarily introduced.

[0098] Taking the target time length of 2 seconds and the first eye movement sampling rate of 120Hz as an example, the identification of the gaze object hit by the user's line of sight can be performed every 100 milliseconds, that is, 12 frames of eye movement data are collected in time sequence (240 frames of eye movement data are collected in 2 seconds, so 12 frames of eye movement data are collected every 100 milliseconds as a group), that is, three-dimensional ray detection is performed according to the 12 frames of eye movement data to obtain a gaze object (the identity ID of the gaze object can be recorded in this embodiment); in this way, 20 gaze object IDs are obtained from 240 frames of eye movement data.

[0099] Suppose 20 gaze object IDs are gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 2, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 3, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, and gaze object 2. Then, the gaze ratio of gaze object 1 = 17 (the number of times of gazing at gaze object 1) / 20 (the total number of times of gazing) = 0.85, the gaze ratio of gaze object 2 = 2 (the number of times of gazing at gaze object 2) / 20 = 0.1, and the gaze ratio of gaze object 3 = 1 (the number of times of gazing at gaze object 3) / 20 = 0.05.

[0100] Taking the preset ratio threshold of 0.8 (corresponding to the preset time threshold of 1600 milliseconds) as an example, since the gaze ratio of gaze object 1 reaches the preset ratio threshold, it can be determined that the 240 frames of eye movement data meet condition 1.

[0101] Next, the standard deviation of the distribution of the user's gaze point in the x direction is calculated according to the x component of the normalized screen coordinates of the plurality of frames of eye movement data, and the standard deviation of the distribution of the user's gaze point in the y direction is calculated according to the y component of the normalized screen coordinates of the plurality of frames of eye movement data. If the standard deviation of the distribution in the x direction and the standard deviation of the distribution in the y direction are both less than a preset standard deviation threshold (for example, the standard deviation threshold is 5% of the width of the display screen), it is determined that the 240 frames of eye movement data meet condition 2.

[0102] Since the 240 frames of eye movement data meet both condition 1 and condition 2, the three-dimensional model of gaze object 1 can be determined as the target three-dimensional model. Next, the model data of the target three-dimensional model is obtained, and the gaze point position is determined according to the 240 frames of eye movement data and the model data.

[0103] In one possible implementation, the process of "determining the gaze point position according to the plurality of frames of eye movement data and the model data" can include:

[0104] Each frame of eye movement data in the plurality of frames of eye movement data is processed into a ray equation of a user's line of sight to obtain a plurality of ray equations;

[0105] A hierarchical bounding volume tree is constructed according to the model data, wherein a plurality of vertices of the target three-dimensional model are connected to form a plurality of polygons, and each leaf node of the hierarchical bounding volume tree represents a polygon;

[0106] For each ray equation in the plurality of ray equations: collision detection is performed according to the ray equation and the hierarchical bounding volume tree to obtain a leaf node of the hierarchical bounding volume tree that intersects the ray equation, and the polygon represented by the leaf node is taken as the polygon corresponding to the ray equation; to obtain the polygons corresponding to the plurality of ray equations, respectively;

[0107] determining, from the plurality of polygons respectively corresponding to the plurality of ray equations, a polygon closest to the distance of the eye movement data acquisition device as a target polygon;

[0108] determining, according to the model data, coordinate data of each vertex of the target polygon;

[0109] determining, according to the coordinate data of each vertex of the target polygon, coordinate data of a center point of the target polygon as the gaze point position, wherein the center point is respectively equal in distance to each vertex.

[0110] Specifically, for each frame of eye movement data in the plurality of frames of eye movement data, the embodiment can obtain a ray equation in the following manner.

[0111] First, the normalized screen coordinates of the frame of eye movement data are converted to a standardized coordinate system defined by a graphic API (Application Programming Interface), and the conversion formula is as follows Formula (1) to map to a unified mathematical space for subsequent calculation.

[0112] Formula (1);

[0113] wherein, x component of the normalized screen coordinates of the frame of eye movement data, x component of the normalized screen coordinates of the frame of eye movement data, y component of the normalized screen coordinates of the frame of eye movement data, y component of the normalized screen coordinates of the frame of eye movement data, z coordinate value of the normalized screen coordinates of the frame of eye movement data, = 1 indicates a fixed direction to the far clipping plane.

[0114] Then, the coordinates in the standardized coordinate system are back-propagated to the three-dimensional world space using the projection matrix and the view matrix of the camera (i.e., the camera in the eye movement tracking device) to obtain a ray equation, and the specific process can refer to the following Formula (2) - Formula (4).

[0115] Formula (2);

[0116] Formula (3);

[0117] Formula (4);

[0118] wherein, represents a starting point of the ray, i.e. coordinates of the camera in the three-dimensional world space; represents a view matrix, represents a projection matrix; represents a direction vector of the ray, i.e. a target direction to which the user's line of sight points; represents a scalar parameter for controlling a distance of a point on the ray from the starting point; represents the point on the ray.

[0119] The above formulas (1)-(4) are used to calculate multiple frames of eye movement data, i.e. multiple ray equations are obtained.

[0120] Then, a bounding volume hierarchy (BVH) can be constructed according to the model data. The BVH is a tree data structure used to accelerate collision detection between rays and object in a three-dimensional scene. The core idea of the BVH is to decompose a complex geometric body into multiple nested bounding boxes (such as cuboids, spheres), and to quickly exclude areas that are not likely to intersect with the ray by checking the intersection of the bounding boxes layer by layer, thereby reducing the amount of calculation.

[0121] Optionally, the process of constructing the BVH can include: determining a minimum axis-aligned bounding box of the target three-dimensional model according to the model data, selecting a midpoint of the longest axis of the minimum axis-aligned bounding box, dividing the target three-dimensional model into left and right two parts to obtain a left subset 1 composed of polygons in the left part and a right subset 1 composed of polygons in the right part; then determining a minimum axis-aligned bounding box of the left subset 1 according to the model data, selecting a midpoint of the longest axis of the minimum axis-aligned bounding box, dividing the left subset 1 into left and right two parts to obtain a left subset 11 and a right subset 12, and similarly dividing the right subset 1 to obtain a left subset 21 and a right subset 22; and further dividing the left subset 11, the right subset 12, the left subset 21 and the right subset 22 according to the above process until each polygon of the target three-dimensional model corresponds to a sub-node (i.e. a leaf node) individually.

[0122] Then, for each ray equation in the multiple ray equations, collision detection can be performed according to the ray equation and the BVH to obtain a leaf node in the BVH that intersects with the ray equation, and the polygon represented by the leaf node is taken as the polygon corresponding to the ray equation.

[0123] Specifically, the collision detection can be performed layer by layer in the order of the BVH from the root node to the leaf node. If the ray equation intersects with the left subset 1, all sub-nodes of the left subset 1 are recursively detected until the leaf node. If the ray equation intersects with the right subset 1, all sub-nodes of the right subset 1 are recursively detected until the leaf node.

[0124] For example, in the ray equation and the collision detection of the leaf node, the following formula (5) can be used for detection.

[0125] Formula (5);

[0126] wherein, , and represent the coordinates of the three vertices of the triangle represented by the leaf node, and the value acquisition process can be referred to below; , and are values to be solved by formula (5), and satisfy .

[0127] Here, , and calculation process can include: first, in the manner of the foregoing step S103, that is, by querying the vertex coordinates in the vertex buffer through the index in the index buffer, obtaining , and corresponding local coordinates, respectively, , and ; Since the ray equation is in the three-dimensional world space, it is necessary to convert , and corresponding local coordinates to world coordinates to obtain , and corresponding world coordinates , and , as in the above formula (5), as in the above formula (5), as in the above formula (5).

[0128] wherein, the process of converting , and corresponding local coordinates to world coordinates is as follows formula (6).

[0129] Formula (6);

[0130] wherein, is a 4x4 matrix containing the position and pose information of the target three-dimensional model, and is a known matrix.

[0131] If the coordinates of the three vertices of a triangle corresponding to a leaf node are substituted into the above formula (5) and have a solution, i.e., the values of , and can be obtained, the triangle represented by the leaf node is determined as the triangle corresponding to the ray equation.

[0132] For each ray equation, the above process is performed to obtain a plurality of polygons corresponding to the plurality of ray equations respectively.

[0133] Since the human eye views the closest polygon to the human eye (or camera) when gazing at the target three-dimensional model, the closest polygon to the distance of the eye movement data acquisition device can be determined from the plurality of polygons corresponding to the plurality of ray equations respectively, as the target polygon.

[0134] Still taking the polygon as a triangle as an example, as introduced in the foregoing, each ray equation substituted into formula (5) can obtain a set of , and , and a plurality of ray equations can obtain a plurality of sets of , and , the embodiment can determine a ray equation that makes minimum, and the triangle corresponding to the ray equation is the closest triangle to the camera, i.e., the target triangle.

[0135] Next, the embodiment can obtain the coordinate data of each vertex of the target polygon according to the model data of the target three-dimensional model, such as the world coordinates , and in the foregoing.

[0136] Finally, the coordinate data of the center point of the target polygon can be determined as the gaze point position according to the coordinate data of each vertex of the target polygon, wherein the distance between the center point and each vertex is equal. Here, the gaze point position is , wherein represents the gaze point position, represents the x component of the world coordinates of the gaze point, represents the y component of the world coordinates of the gaze point, represents the z component of the world coordinates of the gaze point.

[0137] In summary, the embodiment of the present application can convert the user's line of sight into a ray equation from the camera to the gaze point position, combined with the collision detection of the hierarchical bounding box tree, the target polygon hit by the user's line of sight can be accurately identified, and then the center point position of the target polygon is taken as the gaze point position, which is convenient for subsequent calculation of the focal depth.

[0138] In some embodiments of the present application, two modes can be set, namely high-precision acquisition mode and low-power monitoring mode. In the high-precision acquisition mode, eye movement data can be collected at a higher eye movement sampling rate (i.e. the first eye movement sampling rate described above) and the focal depth is determined according to the present application; in the low-power monitoring mode, eye movement data can be collected at a lower eye movement sampling rate (i.e. the second eye movement sampling rate described below), but the focal depth is not determined (only the gaze object ID is recorded).

[0139] Specifically, in the case where the distribution standard deviation is less than the standard deviation threshold, the three-dimensional model of the target gaze object is taken as the target three-dimensional model, the model data of the target three-dimensional model is obtained, and at the same time, the eye movement data of the user is collected at a preset first eye movement sampling rate to obtain multiple frames of eye movement data collected within the target time length; in the case where the distribution standard deviation is greater than or equal to the standard deviation threshold, or there is no target gaze object, the three-dimensional model of the target gaze object is not taken as the target three-dimensional model, the model data of the target three-dimensional model is not obtained, and the eye movement data of the user is continuously collected at a second eye movement sampling rate, wherein the second eye movement sampling rate is less than the first eye movement sampling rate.

[0140] In summary, in the case where the user effectively gazes at the target gaze object (i.e. both conditions 1 and 2 described above are met), the high-precision acquisition mode can be executed, and the full-process processing is performed in this mode to obtain the focal depth. According to experiments, the CPU (Central Processing Unit) occupancy rate in this mode is 30%, and the GPU (Graphics Processing Unit) occupancy rate is 40%; in the case where the user does not effectively gaze at any object (i.e. conditions 1 and / or 2 described above are not met), the low-power monitoring mode can be executed, and the focal depth is not calculated in this mode, only the gaze object ID is recorded, the CPU occupancy rate is less than or equal to 5%, and the GPU is not activated, thereby reducing resource consumption and improving resource utilization.

[0141] In another possible implementation, considering that when the plurality of frames of eye movement data is collected, the problem of rapid eye movement may exist, in order to avoid the interference of rapid eye movement, the process of "collecting eye movement data of the user at a preset first eye movement sampling rate to obtain the plurality of frames of eye movement data collected within a target time length" in the embodiment can include: collecting eye movement data at the first eye movement sampling rate, after collecting each frame of eye movement data, determining whether the frame displacement between the frame of eye movement data and the previous frame of eye movement data is less than a preset frame displacement threshold; if yes, controlling the timer to continue timing, and meanwhile continuing to collect eye movement data, until the timing time length of the timer reaches the target time length, and the plurality of frames of eye movement data are obtained; if no, controlling the timer to reset to 0 and then continue timing, and meanwhile continuing to collect eye movement data, until the timing time length of the timer reaches the target time length, and the plurality of frames of eye movement data are obtained.

[0142] For the process of collecting eye movement data of the user at the second eye movement sampling rate, reference can be made to the above-mentioned collection process at the first eye movement sampling rate, which will not be repeated here.

[0143] That is, in the process of collecting eye movement data, the embodiment can detect the frame displacement between adjacent two frames of eye movement data in real time, when the frame displacement is greater than or equal to the frame displacement threshold, it indicates that the user has the condition of rapid eye movement, at this time, it is considered that the previously collected eye movement data used for subsequent focal depth calculation will cause the accuracy of focal depth to decrease, then the timer is reset, and the focal depth is calculated according to the newly collected eye movement data after the reset, so as to improve the accuracy of the focal depth.

[0144] The embodiment can exclude the interference of rapid eye movement by means of adjacent frame displacement, improve the accuracy of focal depth, and further improve the rendering effect of the three-dimensional scene and the viewing experience of the user.

[0145] In some embodiments of the present application, the process of the above step S104 "performing vertex density analysis on the target vertex set to obtain the focal depth for rendering the three-dimensional scene" is introduced.

[0146] Considering that the focal depth is directly obtained from the gaze point position, there may be a problem that the subtle fluctuations in the high-density area are ignored. There may also be high-density areas and sparse areas in the sphere. In order to make the high-density area present a more optimal rendering effect to the human eye, optionally, the embodiment can perform vertex density clustering processing on the target vertex set to obtain a target vertex cluster with the most number of vertices, and then perform vertex density weighting processing according to the target vertex cluster to obtain the focal depth.

[0147] Optionally, the process of "performing vertex density clustering on the target vertex set to obtain a target vertex cluster with the largest number of vertices" may include: determining the number of vertices in the target vertex set as the number of vertices in the sphere; determining the density of the sphere based on the number of vertices in the sphere; obtaining a clustering radius based on the density of the sphere, wherein the clustering radius and the density of the sphere are inversely correlated; clustering the target vertex set based on the clustering radius to obtain several vertex clusters; and determining the vertex cluster with the largest number of vertices from the several vertex clusters as the target vertex cluster.

[0148] Optionally, the process of “determining the density of the target sphere based on the number of vertices within the sphere” can adopt the following formula (7).

[0149] Formula (7);

[0150] in, represents the number of vertices in the sphere, represents the target value, represents the density of the sphere.

[0151] It should be noted that, in this embodiment, the target value It can be determined according to the actual scene. For example, if the overall accuracy of the scene is high, the To improve the sensitivity, it can be expanded to 2mm. If the scene is rough, it can be expanded to 10mm to avoid missed detection.

[0152] Of course, the above 2mm and 10mm are only examples and are not intended to limit the present application.

[0153] Alternatively, the following formula (8) can be used to obtain the cluster radius based on the density of the spheres.

[0154] Formula (8);

[0155] in, represents the cluster radius; Indicates the base radius, which is the default value.

[0156] It can be seen from formula (8) that the higher the density of the fixation area, the smaller the clustering radius, which can better focus on micro details; the lower the density of the fixation area, the larger the clustering radius, which can better cover the macro structure.

[0157] Optionally, this embodiment can use the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm based on the cluster radius The target vertex set is clustered to obtain a plurality of vertex clusters, and then a vertex cluster with the largest number of vertices is determined from the plurality of vertex clusters as a target vertex cluster.

[0158] Optionally, the process of obtaining the focal depth according to the target vertex cluster can include: generating a minimum bounding box of the target vertex cluster, determining a volume of the minimum bounding box; determining a number of vertices contained in the target vertex cluster; determining a local density according to the number of vertices contained in the target vertex cluster and the volume of the minimum bounding box; obtaining a depth value of each vertex in the target vertex cluster from position data of each vertex in the target vertex cluster; and performing weighted average according to the local density and the depth value of each vertex in the target vertex cluster to obtain the focal depth.

[0159] Here, the minimum bounding box of the target vertex cluster can be a sphere bounding box, a cuboid bounding box, or other bounding boxes, and the embodiment is not limited in particular.

[0160] Optionally, the number of vertices contained in the target vertex cluster can be divided by the volume of the minimum bounding box to obtain a quotient as the local density. .

[0161] In the embodiment, the position data of each vertex in the target vertex cluster can refer to the calculation process of the foregoing 、 and , and will not be described here again.

[0162] Since the vertices in the target vertex cluster can be at different depths and scales, in order to unify the analysis, the position data of each vertex in the target vertex cluster can be converted to a local coordinate system with the gaze point position as the coordinate origin.

[0163] Specifically, taking the jth vertex in the target vertex cluster as an example, the position data of the jth vertex is denoted as , and the conversion of the position data of the jth vertex to the local coordinate system is as follows.

[0164] Formula (9);

[0165] wherein represents the coordinate of the jth vertex in the target vertex cluster in the local coordinate system, represents the coordinate of the jth vertex in the target vertex set in the world coordinate system.

[0166] Then, for the jth vertex in the target vertex cluster, the coordinate of the jth vertex in the local coordinate system is the Z component in the Z component of the jth vertex is extracted as the depth value of the jth vertex Similarly, the depth values of other vertices can be obtained.

[0167] Optionally, the process of obtaining the focal depth according to the local density and the depth values of all vertices in the target vertex cluster can use the following formula (10).

[0168] Formula (10);

[0169] wherein, denotes the focal depth, denotes the number of vertices included in the target vertex cluster.

[0170] Compared with the clustering method of the traditional DBSCAN clustering algorithm using a fixed radius, the clustering radius based on the vertex density is provided in the embodiment, which can better distinguish the vertex density difference of different regions of the target three-dimensional model, so that the focal depth obtained is more in line with the detail distribution, so that the high-density region can have a better rendering effect in the human eye.

[0171] In some embodiments of the present application, after obtaining the focal depth, the depth of field parameter value can also be determined according to the focal depth, and the three-dimensional scene can be rendered according to the depth of field parameter value and the focal depth.

[0172] Optionally, the process of determining the depth of field parameter value according to the focal depth can include: for each pixel of the three-dimensional scene, determining the depth difference corresponding to the pixel according to the pixel depth of the pixel and the focal depth, to obtain the depth differences corresponding to all pixels of the three-dimensional scene respectively; dividing the three-dimensional scene into multiple rendering regions according to the depth differences corresponding to all pixels of the three-dimensional scene respectively, wherein different rendering regions use different rendering technologies; determining the depth of field parameter value of each pixel of the three-dimensional scene according to the depth difference corresponding to the pixel and the rendering region where the pixel is located, to obtain the depth of field parameter values corresponding to all pixels of the three-dimensional scene respectively.

[0173] Optionally, the process of determining the depth difference corresponding to the pixel according to the pixel depth of the pixel and the focal depth can use the following formula (11).

[0174] Formula (11);

[0175] wherein, denotes the pixel depth of the pixel in the three-dimensional scene, denotes the depth difference corresponding to the pixel in the three-dimensional scene.

[0176] Optionally, for each pixel in the three-dimensional scene, when the pixel satisfies ≤1cm, the pixel is divided into the focus core area, when the pixel satisfies 1cm ≤5cm, the pixel is divided into the transition area, when the pixel satisfies >5cm, the pixel is divided into the edge area.

[0177] Optionally, for the focus core area, a high-precision PBR (Physically Based Rendering) material and Real-Time Ray Tracing (RTX) rendering technology can be used, and the performance ratio is 60%; for the transition area, a medium-precision PBR material and Screen Space Reflection (SSR) rendering technology can be used, and the performance ratio is 30%; for the edge area, a low-mode, pre-baked lighting, and two-dimensional parallax map rendering technology can be used, and the performance ratio is 10%.

[0178] Optionally, the foreground blur radius and the background blur radius of each pixel in the three-dimensional scene can be calculated according to the following formula (12) and formula (13).

[0179] Formula (12);

[0180] Formula (13);

[0181] wherein, represents the foreground blur radius of the pixel in the three-dimensional scene, represents the background blur radius of the pixel in the three-dimensional scene, and is an artistic control coefficient (by default ).

[0182] Optionally, for each pixel in the three-dimensional scene, if the rendering area where the pixel is located is the focus core area, the blur kernel R of the pixel is 0 (i.e. no blur), if the rendering area where the pixel is located is the transition area, the blur kernel R of the pixel is any value in the range of 3-7 (which value to use can be determined according to the actual scene), if the rendering area where the pixel is located is the edge area, the blur kernel R of the pixel is any value in the range of 10-15 (which value to use can be determined according to the actual scene).

[0183] Then the above foreground blur radius, background blur radius and blur kernel are the depth of field parameter values in this embodiment.

[0184] In summary, the embodiment can not only generate a focal depth dynamically adjusted according to the user's line of sight, but also generate a depth of field parameter value dynamically adjusted according to the user's line of sight. Based on the focal depth and the depth of field parameter value dynamically adjusted according to the user's line of sight, the rendered three-dimensional scene can have better detailed perception effect in the high-density area of the user's gaze point, and the rendering effect of the three-dimensional scene and the user's viewing experience are improved.

[0185] The above introduces a focal depth determination method provided by the embodiment of the application. The following will introduce a device for executing the focal depth determination method.

[0186] Please refer to Figure 2 , Figure 2 The structure diagram of a focal depth determination device provided by the embodiment of the application. As Figure 2 shown, the focal depth determination device can include:

[0187] The eye movement data acquisition module 201 is configured to collect eye movement data of a user in a process of viewing a three-dimensional scene according to a preset first eye movement sampling rate, to obtain a plurality of frames of eye movement data collected in a target time length.

[0188] The gaze point position determination module 202 is configured to determine a gaze point position of a user's line of sight in a target three-dimensional model according to the plurality of frames of eye movement data, wherein the target three-dimensional model refers to a three-dimensional model gazed at by the user in the three-dimensional scene, and the target three-dimensional model includes a plurality of vertices.

[0189] The sphere-in-vertex determination module 203 is configured to generate a sphere with a radius of a target value with the gaze point position as a sphere center, and obtain a vertex set in the target three-dimensional model located in the sphere as a target vertex set, wherein the target vertex set includes position data of each of the plurality of vertices.

[0190] The focal depth determination module 204 is configured to perform vertex density analysis on the target vertex set to obtain a focal depth for rendering the three-dimensional scene.

[0191] In a possible implementation, when determining the gaze point position of the user's line of sight in the target three-dimensional model according to the plurality of frames of eye movement data, the gaze point position determination module can be specifically configured to:

[0192] divide the plurality of frames of eye movement data into a plurality of groups in chronological order, determine a gazed object corresponding to each group according to the eye movement data in each group, and obtain a plurality of gazed objects corresponding to the plurality of groups respectively;

[0193] determine whether there is a target gazed object with a gazing ratio reaching a preset ratio threshold according to the plurality of gazed objects corresponding to the plurality of groups respectively, wherein the gazing ratio of a gazed object is a ratio of a gazing number of the gazed object to a total gazing number.

[0194] If yes, a distribution standard deviation of the gaze points of the user is calculated according to the multi-frame eye movement data;

[0195] If the distribution standard deviation is less than a preset standard deviation threshold, a three-dimensional model of the target gaze object is taken as a target three-dimensional model, and model data of the target three-dimensional model is obtained;

[0196] The gaze point position is determined according to the multi-frame eye movement data and the model data.

[0197] In a possible implementation, when determining the gaze point position according to the multi-frame eye movement data and the model data, the gaze point position determination module can be specifically configured to:

[0198] process each frame of eye movement data in the multi-frame eye movement data into a ray equation of a user's line of sight, to obtain a plurality of ray equations;

[0199] construct a hierarchical bounding box tree according to the model data, wherein a plurality of vertices of the target three-dimensional model are connected to form a plurality of polygons, and each leaf node of the hierarchical bounding box tree represents a polygon;

[0200] for each ray equation in the plurality of ray equations: perform collision detection according to the ray equation and the hierarchical bounding box tree, to obtain a leaf node of the hierarchical bounding box tree that intersects with the ray equation, and take the polygon represented by the leaf node as a polygon corresponding to the ray equation; and obtain the polygons corresponding to the plurality of ray equations respectively;

[0201] determine, from the polygons corresponding to the plurality of ray equations respectively, a polygon closest to the distance of the eye movement data acquisition device, as a target polygon;

[0202] determine, according to the model data, coordinate data of each vertex of the target polygon;

[0203] determine, according to the coordinate data of each vertex of the target polygon, coordinate data of a center point of the target polygon as the gaze point position, wherein the center point is equal in distance to each vertex.

[0204] In a possible implementation, the focal point depth determination apparatus provided by the present application can further include a mode adjustment module.

[0205] The mode adjusting module is configured to, when the distribution standard deviation is less than the standard deviation threshold, return to collect eye movement data of the user at a preset first eye movement sampling rate to obtain a plurality of frames of eye movement data collected within a target time length while performing acquisition of model data of a three-dimensional model of a target gaze object as a target three-dimensional model; and when the distribution standard deviation is greater than or equal to the standard deviation threshold or there is no target gaze object, not performing acquisition of model data of a three-dimensional model of a target gaze object as a target three-dimensional model and continuing to collect eye movement data of the user at a second eye movement sampling rate, wherein the second eye movement sampling rate is less than the first eye movement sampling rate.

[0206] In a possible implementation, the focal depth determination module, when performing vertex density analysis on the target vertex set to obtain the focal depth for rendering the three-dimensional scene, can be specifically configured to:

[0207] perform vertex density clustering processing on the target vertex set to obtain a target vertex cluster with the largest number of vertices;

[0208] perform vertex density weighted processing on the target vertex cluster to obtain the focal depth.

[0209] In a possible implementation, the focal depth determination module, when performing vertex density clustering processing on the target vertex set to obtain a target vertex cluster with the largest number of vertices, can be specifically configured to:

[0210] determine the number of vertices in the target vertex set as the number of vertices in the sphere;

[0211] determine the density of the sphere according to the number of vertices in the sphere;

[0212] obtain a clustering radius according to the density of the sphere, wherein the clustering radius and the density of the sphere are inversely related;

[0213] perform clustering on the target vertex set based on the clustering radius to obtain a plurality of vertex clusters;

[0214] determine a vertex cluster with the largest number of vertices from the plurality of vertex clusters as the target vertex cluster.

[0215] In a possible implementation, the focal depth determination module, when performing vertex density weighted processing on the target vertex cluster to obtain the focal depth, can be specifically configured to:

[0216] generate a minimum bounding box of the target vertex cluster and determine the volume of the minimum bounding box;

[0217] determine the number of vertices contained in the target vertex cluster;

[0218] determine a local density according to the number of vertices contained in the target vertex cluster and the volume of the minimum bounding box;

[0219] obtaining the depth value of each vertex in the target vertex cluster from the position data of each vertex in the target vertex cluster;

[0220] performing weighted average according to the local density and the depth value of each vertex in the target vertex cluster to obtain the focal point depth.

[0221] The modules in the focal point depth determination apparatus can be implemented by software, hardware or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be invoked and executed by the processor to perform the operations corresponding to the modules.

[0222] The electronic device can comprise at least one processor and a memory connected to the processor, wherein:

[0223] The memory is configured to store a computer program.

[0224] The processor is configured to execute the computer program, so that the electronic device can implement any of the focal point depth determination methods provided in the embodiments of the present application.

[0225] Reference Figure 3 The electronic device can include, but is not limited to, a fixed terminal such as a mobile phone, a notebook computer, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a desktop computer, etc. Figure 3 The electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0226] As shown in Figure 3 The electronic device can include a processing device (such as a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or programs loaded from a storage device 708 into a random access memory (RAM) 703. In the state that the electronic device is powered on, the RAM 703 also stores various programs and data required for the operation of the electronic device. The processing device 701, the ROM 702 and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0227] In general, the following devices can be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 708 including, for example, a memory card, a hard disk, and the like; and communication devices 709. The communication devices 709 can allow the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device having various devices is shown, but it is understood that all of the shown devices are not required to be implemented or present. More or less devices can alternatively be implemented or present.

[0228] The embodiments of the present application also provide a computer program product including computer readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the focus depth determination methods provided by the embodiments of the present application.

[0229] The embodiments of the present application also provide a computer readable storage medium carrying one or more computer programs, which, when executed by an electronic device, can cause the electronic device to implement any of the focus depth determination methods provided by the embodiments of the present application.

[0230] In addition, it should be noted that the apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments. In addition, the connection relationship between the modules in the apparatus embodiments provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0231] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special integrated circuit, special CPU, special memory, special component, etc. Generally, any function completed by computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, software program implementation is a better embodiment. Based on such understanding, the technical solution of the application or the part of the application which makes contribution to the prior art can be embodied in the form of software product, which is stored in readable storage medium, such as computer floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a plurality of instructions for making a computer device (which can be personal computer, training device or network device, etc.) execute the method described in various embodiments of the application.

[0232] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be achieved in the form of a computer program product, entirely or partially.

[0233] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiments of the application is generated entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as training device, data center, etc. integrated with one or more available media sets. The available medium can be magnetic medium (such as floppy disk, hard disk, magnetic tape), optical medium (such as DVD) or semiconductor medium (such as solid state disk (SSD)) etc.

Claims

1. A method for determining focal depth, characterized in that: include: While the user is viewing the three-dimensional scene, collecting eye movement data of the user according to a preset first eye movement sampling rate to obtain multiple frames of eye movement data collected within a target duration; Determining, based on the multiple frames of eye movement data, a location of a gaze point of the user in a target three-dimensional model, wherein the target three-dimensional model refers to a three-dimensional model that the user is gazing at in the three-dimensional scene, and the target three-dimensional model includes a plurality of vertices; Generate a sphere with a target radius and a sphere centering on the gaze point, and obtain a vertex set in the target three-dimensional model that is within the sphere as a target vertex set, wherein the target vertex set includes position data of a plurality of vertices; Vertex density analysis is performed on the target vertex set to obtain a focal depth for rendering the three-dimensional scene.

2. The method for determining focal depth according to claim 1, wherein: Determining the gaze point position of the user's line of sight in the target three-dimensional model based on the multiple frames of eye movement data includes: Dividing the multiple frames of eye movement data into a plurality of groups according to the order of acquisition time, and determining the gaze object corresponding to each group based on the eye movement data in each group, so as to obtain the gaze objects corresponding to the plurality of groups respectively; According to the gaze objects corresponding to the plurality of groups, determining whether there is a target gaze object whose gaze ratio reaches a preset ratio threshold, wherein the gaze ratio of a gaze object is a ratio of the number of gazes of the gaze object to the total number of gazes; If yes, calculating the distribution standard deviation of the user's gaze point based on the multiple frames of eye movement data; If the distribution standard deviation is less than a preset standard deviation threshold, taking the three-dimensional model of the target gaze object as the target three-dimensional model, and acquiring model data of the target three-dimensional model; The gaze point position is determined according to the multiple frames of eye movement data and the model data.

3. The method for determining focal depth according to claim 2, wherein: The determining the gaze point position according to the multiple frames of eye movement data and the model data includes: Processing each frame of eye movement data in the multiple frames of eye movement data into a ray equation of the user's line of sight to obtain multiple ray equations; Constructing a hierarchical bounding box tree according to the model data, wherein a plurality of vertices of the target three-dimensional model are connected to form a plurality of polygons, and each leaf node of the hierarchical bounding box tree represents one of the polygons; For each of the multiple ray equations: performing collision detection based on the ray equation and the hierarchical bounding box tree, obtaining a leaf node in the hierarchical bounding box tree that intersects with the ray equation, and using the polygon represented by the leaf node as the polygon corresponding to the ray equation; thereby obtaining polygons corresponding to the multiple ray equations respectively; Determine, from the polygons corresponding to the multiple ray equations, a polygon closest to the eye movement data acquisition device as a target polygon; Determine coordinate data of each vertex of the target polygon according to the model data; According to the coordinate data of each vertex of the target polygon, the coordinate data of the center point of the target polygon is determined as the gaze point position, wherein the center point is equidistant from each vertex.

4. The method for determining focal depth according to claim 2, wherein: Also includes: When the distribution standard deviation is less than the standard deviation threshold, while performing the step of using the three-dimensional model of the target gaze object as the target three-dimensional model and obtaining model data of the target three-dimensional model, returning to the step of collecting the user's eye movement data at the preset first eye movement sampling rate to obtain multiple frames of eye movement data collected within a target duration; When the distribution standard deviation is greater than or equal to the standard deviation threshold, or the target gaze object does not exist, the step of using the three-dimensional model of the target gaze object as the target three-dimensional model is not performed, the model data of the target three-dimensional model is obtained, and the eye movement data of the user is continued to be collected according to the second eye movement sampling rate, wherein the second eye movement sampling rate is less than the first eye movement sampling rate.

5. The method for determining focal depth according to claim 1, wherein: The performing vertex density analysis on the target vertex set to obtain a focal depth for rendering the three-dimensional scene includes: Performing vertex density clustering on the target vertex set to obtain a target vertex cluster with the largest number of vertices; Vertex density weighting processing is performed according to the target vertex cluster to obtain the focal depth.

6. The method for determining focal depth according to claim 5, wherein: The vertex density clustering process is performed on the target vertex set to obtain a target vertex cluster with the largest number of vertices, including: Determine the number of vertices in the target vertex set as the number of vertices in the sphere; Determining the density of the sphere according to the number of vertices in the sphere; Obtaining a cluster radius according to the density of the spheres, wherein the cluster radius is inversely correlated with the density of the spheres; Clustering the target vertex set based on the clustering radius to obtain a plurality of vertex clusters; A vertex cluster with the largest number of vertices is determined from the plurality of vertex clusters as the target vertex cluster.

7. The method for determining focal depth according to claim 5, wherein: The performing vertex density weighted processing according to the target vertex cluster to obtain the focal depth includes: Generating a minimum bounding box of the target vertex cluster, and determining the volume of the minimum bounding box; Determine the number of vertices included in the target vertex cluster; Determining a local density according to the number of vertices included in the target vertex cluster and the volume of the minimum bounding box; Obtaining respective depth values ​​of all vertices in the target vertex cluster from respective position data of all vertices in the target vertex cluster; The focal depth is obtained by performing a weighted average based on the local density and the depth values ​​of all vertices in the target vertex cluster.

8. A focal depth determination device, characterized in that: include: An eye movement data acquisition module is used to collect eye movement data of the user according to a preset first eye movement sampling rate while the user is viewing a three-dimensional scene, so as to obtain multiple frames of eye movement data collected within a target duration; a gaze point position determination module, configured to determine a gaze point position of the user's line of sight in a target three-dimensional model based on the multiple frames of eye movement data, wherein the target three-dimensional model refers to a three-dimensional model that the user is gazing at in the three-dimensional scene, and the target three-dimensional model includes a plurality of vertices; a module for determining vertices within a sphere, configured to generate a sphere with a radius of a target value with the gaze point position as the sphere center, and obtain a set of vertices within the sphere in the target three-dimensional model as a target vertex set, wherein the target vertex set includes position data of respective vertices; A focus depth determination module is used to perform vertex density analysis on the target vertex set to obtain a focus depth for rendering the three-dimensional scene.

9. An electronic device, characterized in that: comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program so as to enable the electronic device to implement the method for determining depth of focus according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the method for determining depth of focus as described in any one of claims 1 to 7.

Images