A focus depth determination method and related apparatus

Abstract

The application discloses a focal depth determination method and related device, and relates to the technical field of rendering, and comprises the following steps: in the process that a user watches a three-dimensional scene, collecting eye movement data of the user according to a preset eye movement sampling rate to obtain a plurality of frames of eye movement data collected in a target time length; determining a gaze point position of a user sight line in a target three-dimensional model according to the plurality of frames of eye movement data; generating a sphere with a target value as a radius with the gaze point position as a sphere center; obtaining a target vertex set located in the sphere in the target three-dimensional model; performing vertex density analysis on the target vertex set to obtain a focal depth used for rendering the three-dimensional scene. The application can generate a focal depth dynamically adjusted according to the user sight line, improves the detail resolution capability of the user at the gaze point, and further improves the watching experience of the user on the three-dimensional scene.

Description

Technical Field

[0001] This application relates to the field of rendering technology, and in particular to a method and related apparatus for determining focus depth. Background Technology

[0002] Currently, 3D models in virtual reality (VR) scenarios (such as VR game scenarios and guided VR tour scenarios) are usually rendered offline. However, discrete rendering uses a fixed-focus rendering method, which results in the blurred and unblurred areas in the rendered 3D model remaining unchanged and unable to be dynamically adjusted according to the user's line of sight, thus reducing the user's viewing experience of the 3D model. Summary of the Invention

[0003] In view of the above problems, this application provides a method and related apparatus for determining focal depth, so as to achieve the purpose of generating a focal depth that dynamically adjusts according to the user's line of sight. The specific solution is as follows:

[0004] The first aspect of this application provides a method for determining focus depth, including:

[0005] During the user's viewing of the 3D scene, eye movement data of the user is collected according to the preset first eye movement sampling rate to obtain multiple frames of eye movement data collected within the target duration;

[0006] Based on the multi-frame eye-tracking data, the position of the user's gaze point in the target 3D model is determined, wherein the target 3D model refers to the 3D model that the user gazes at in the 3D scene, and the target 3D model includes several vertices;

[0007] A sphere with a radius of a target value is generated with the gaze point position as the center. The set of vertices located within the sphere in the target 3D model is obtained as the target vertex set, wherein the target vertex set includes the position data of each of the multiple vertices.

[0008] Vertex density analysis is performed on the target vertex set to obtain the focal depth used for rendering the 3D scene.

[0009] In one possible implementation, determining the position of the user's gaze point in the target 3D model based on the multi-frame eye-tracking data includes:

[0010] The multi-frame eye-tracking data is divided into multiple groups according to the acquisition time sequence. Based on the eye-tracking data in each group, the gaze object corresponding to each group is determined to obtain the gaze objects corresponding to the multiple groups respectively.

[0011] Based on the gaze objects corresponding to the multiple groups, it is determined whether there are target gaze objects whose gaze ratio reaches a preset ratio threshold. The gaze ratio of a gaze object is the ratio of the number of gazes of that gaze object to the total number of gazes.

[0012] If so, then calculate the standard deviation of the user's gaze points based on the multi-frame eye-tracking data;

[0013] If the standard deviation of the distribution is less than the preset standard deviation threshold, then the three-dimensional model of the target object being gazed at is taken as the target three-dimensional model, and the model data of the target three-dimensional model is obtained;

[0014] The fixation point position is determined based on the multi-frame eye-tracking data and the model data.

[0015] In one possible implementation, determining the fixation point position based on the multi-frame eye-tracking data and the model data includes:

[0016] Each frame of eye-tracking data in the multi-frame eye-tracking data is processed into a ray equation for the user's gaze, so as to obtain multiple ray equations;

[0017] A hierarchical bounding box tree is constructed based on the model data, wherein several vertices of the target 3D model are connected to form several polygons, and each leaf node of the hierarchical bounding box tree represents one of the polygons.

[0018] For each of the multiple ray equations: perform collision detection based on the ray equation and the hierarchical bounding box tree to obtain the leaf node in the hierarchical bounding box tree that intersects with the ray equation, and take the polygon represented by the leaf node as the polygon corresponding to the ray equation; thus obtaining the polygons corresponding to the multiple ray equations respectively.

[0019] From the polygons corresponding to the plurality of ray equations, determine the polygon that is closest to the eye-tracking data acquisition device, and use it as the target polygon;

[0020] The coordinate data of each vertex of the target polygon are determined based on the model data;

[0021] Based on the coordinate data of each vertex of the target polygon, the coordinate data of the center point of the target polygon is determined as the position of the gaze point, wherein the center point is equidistant from each vertex.

[0022] One possible implementation also includes:

[0023] When the standard deviation of the distribution is less than the standard deviation threshold, while performing the step of taking the three-dimensional model of the target gaze object as the target three-dimensional model and obtaining the model data of the target three-dimensional model, the step of collecting the user's eye movement data according to the preset first eye movement sampling rate is returned to obtain multiple frames of eye movement data collected within the target duration.

[0024] If the standard deviation of the distribution is greater than or equal to the standard deviation threshold, or if the target gaze object does not exist, the following steps are not performed: using the 3D model of the target gaze object as the target 3D model, acquiring the model data of the target 3D model, and continuing to collect the user's eye movement data according to the second eye movement sampling rate, wherein the second eye movement sampling rate is less than the first eye movement sampling rate.

[0025] In one possible implementation, performing vertex density analysis on the target vertex set to obtain the focal depth for rendering the 3D scene includes:

[0026] Perform vertex density clustering on the target vertex set to obtain the target vertex cluster with the largest number of vertices;

[0027] The focal depth is obtained by performing vertex density weighting processing on the target vertex cluster.

[0028] In one possible implementation, performing vertex density clustering on the target vertex set to obtain the target vertex cluster with the largest number of vertices includes:

[0029] Determine the number of vertices in the target vertex set as the number of vertices inside the sphere;

[0030] The density of the sphere is determined based on the number of vertices within the sphere.

[0031] The cluster radius is obtained based on the density of the spheres, wherein the cluster radius is inversely correlated with the density of the spheres;

[0032] Based on the clustering radius, the target vertex set is clustered to obtain several vertex clusters;

[0033] The vertex cluster with the largest number of vertices among the aforementioned vertex clusters is selected as the target vertex cluster.

[0034] In one possible implementation, the step of performing vertex density weighting processing based on the target vertex cluster to obtain the focal depth includes:

[0035] Generate the minimum bounding box of the target vertex cluster and determine the volume of the minimum bounding box;

[0036] Determine the number of vertices contained in the target vertex cluster;

[0037] The local density is determined based on the number of vertices contained in the target vertex cluster and the volume of the minimum bounding box;

[0038] The depth values ​​of each vertex in the target vertex cluster are obtained from the position data of each vertex in the target vertex cluster.

[0039] The focal depth is obtained by taking a weighted average of the local density and the depth values ​​of all vertices in the target vertex cluster.

[0040] A second aspect of this application provides a focal depth determining device, comprising:

[0041] The eye-tracking data acquisition module is used to collect the user's eye-tracking data according to a preset first eye-tracking sampling rate during the user's viewing of a 3D scene, so as to obtain multiple frames of eye-tracking data collected within a target duration.

[0042] The gaze point location determination module is used to determine the gaze point location of the user's gaze in the target 3D model based on the multi-frame eye-tracking data, wherein the target 3D model refers to the 3D model that the user gazes at in the 3D scene, and the target 3D model includes a number of vertices;

[0043] The sphere-in-body vertex determination module is used to generate a sphere with the gaze point position as the center and the radius as the target value, and to obtain the set of vertices located in the sphere in the target 3D model as the target vertex set, wherein the target vertex set includes the position data of each of the multiple vertices;

[0044] The focus depth determination module is used to perform vertex density analysis on the target vertex set to obtain the focus depth used for rendering the 3D scene.

[0045] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the focus depth determination method of the first aspect or any implementation thereof.

[0046] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0047] The memory is used to store computer programs;

[0048] The processor is used to execute the computer program so that the electronic device can implement the focus depth determination method of the first aspect or any implementation thereof.

[0049] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform the focus depth determination method described in the first aspect or any implementation thereof.

[0050] By employing the aforementioned technical solution, the focal depth determination method provided in this application collects the user's eye movement data according to a preset eye movement sampling rate during the user's viewing of a 3D scene, obtaining multiple frames of eye movement data collected within a target duration. Then, based on these multiple frames of eye movement data, the position of the user's gaze point in the target 3D model is determined. Considering that the vertex density around the gaze point can characterize the model complexity of the gaze point region, and that the more complex the model, the more necessary it is to reduce the focal depth to ensure that planes aligned with the gaze point remain sharp while those not aligned are less sharp, thereby avoiding visual interference from non-aligned planes and improving the user's ability to distinguish details at the gaze point, this application can generate a sphere with the gaze point position as the center and a radius of the target value, obtain the target vertex set located within the sphere in the target 3D model, perform vertex density analysis on the target vertex set, and obtain the focal depth used for rendering the 3D scene. Therefore, this application can generate a focal depth that dynamically adjusts with the user's gaze, improving the user's ability to distinguish details at the gaze point and thus enhancing the user's viewing experience of the 3D scene. Attached Figure Description

[0051] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0052] Figure 1 A flowchart illustrating a method for determining focus depth provided in this application;

[0053] Figure 2 A schematic diagram of a focal depth determining device provided in this application;

[0054] Figure 3 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0055] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0056] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0058] To enable those skilled in the art to better understand this application, the relevant terms used in this application are explained below.

[0059] Depth of focus (DFO): This refers to the distance the focal point can move along the optical axis while maintaining image sharpness. A larger DFO allows for a wider range of out-of-focus areas, but resolution may decrease; a smaller DFO ensures that only the area near the focal plane is sharp, resulting in higher detail resolution.

[0060] In 3D rendering, depth of field is a key parameter controlling the depth-of-field effect. The depth-of-field effect simulates the depth of field in an optical system, keeping specific objects or areas in focus while blurring others, thus enhancing the image's depth and realism. For example, in 3ds Max (a professional 3D modeling, animation, rendering, and special effects software), by adjusting the camera's depth of field and other parameters, one can precisely control which objects are in focus and which are blurred, making the rendered image more artistically expressive.

[0061] This application provides a method for determining focus depth, which can be applied to terminals or servers with data processing capabilities, and can also be applied to systems composed of the aforementioned terminals and servers.

[0062] Of course, the focus depth determination method provided in this application can also be applied to other applications, and this application does not make any specific limitations.

[0063] To enable those skilled in the art to better understand this application, the focal depth determination method of the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0064] Reference Figure 1 , Figure 1This is a flowchart illustrating a method for determining focus depth provided in an embodiment of this application, as shown below. Figure 1 As shown, the method for determining the focal depth may include:

[0065] Step S101: During the user's viewing of the 3D scene, collect the user's eye movement data according to the preset first eye movement sampling rate to obtain multiple frames of eye movement data collected within the target duration.

[0066] This embodiment allows users to wear eye-tracking devices while viewing a 3D scene and sets a first eye-tracking sampling rate, enabling the eye-tracking device to collect the user's eye-tracking data according to the preset first eye-tracking sampling rate.

[0067] Considering the significant errors in single-frame eye-tracking data—for example, users often saccade while viewing 3D scenes, and single-frame eye-tracking data struggles to identify these saccades—the accuracy of focusing depth identification using single-frame data is poor. Therefore, alternatively, an eye-tracking device can be used to collect multiple frames of eye-tracking data over a target duration to more accurately identify focusing depth.

[0068] Optionally, a circular data buffer can be established to dynamically store multiple frames of eye-tracking data within the target duration. For example, with a first eye-tracking sampling rate of 120 Hz and a target duration of 2 seconds, 240 frames of eye-tracking data can be stored in the circular data buffer.

[0069] Of course, the first eye movement sampling rate and target duration mentioned above can be other values, and can be set according to the actual scenario. No specific restrictions are made here.

[0070] Optionally, the eye-tracking device mentioned above can be an eye-tracking device integrated into the head-mounted display, such as Varjo XR-4, HTC Vive Pro Eye, Meta Quest Pro, etc.; alternatively, the eye-tracking device can also be an external eye-tracking device, such as Tobii Eye Tracker 5, etc.

[0071] Step S102: Determine the position of the user's gaze point in the target 3D model based on multi-frame eye-tracking data.

[0072] Understandably, each frame of eye-tracking data collected by an eye-tracking device includes gaze coordinates, confidence level, and timestamp. To eliminate device differences, standardize data scales, simplify computational logic, and provide a standardized basis for subsequent analysis, it is typically necessary to convert the gaze coordinates into normalized screen coordinates. Specifically, the gaze coordinates are normalized to obtain normalized screen coordinates. It is worth noting that normalized screen coordinates are two-dimensional coordinates.

[0073] Therefore, this embodiment can determine the position of the user's gaze point in the target 3D model based on the normalized screen coordinates of multiple frames. In other words, this embodiment can fuse the user's gaze within the target duration into a single gaze point and obtain the position data of the gaze point.

[0074] Here, the 3D model where the gaze point is located is defined as the target 3D model, that is, the target 3D model refers to the 3D model that the user gazes at in the 3D scene.

[0075] It is understandable that in a 3D scene, each object corresponds to a 3D model, and each 3D model includes several vertices. These vertices can be connected to form several polygons (such as triangles). Based on this, the target 3D model includes several vertices. Here, "several" refers to a value greater than or equal to 3, and the specific value is related to the 3D model; this application does not impose specific limitations on it.

[0076] Step S103: Generate a sphere with the target radius and the center of the sphere at the gaze point position, and obtain the set of vertices located inside the sphere in the target 3D model as the target vertex set.

[0077] The target vertex set includes the position data of multiple vertices.

[0078] Considering the close relationship between vertex density and depth of focus in the foveation region, the denser the foveation region, the more necessary it is to reduce the sharpness of the surrounding areas to minimize visual interference and improve the user's ability to distinguish details within the foveation region. Depth of focus narrows the area of ​​sharp imaging (i.e., limits the sharp region), ensuring that only planes strictly aligned with the foveation point maintain high sharpness, while non-aligned planes become naturally blurred due to exceeding the depth of focus range. Based on this, this embodiment can obtain the depth of focus for rendering the target 3D model, i.e., the depth of focus for rendering the 3D scene, by analyzing the vertex density of the foveation region.

[0079] To identify the gaze point region, optionally, a sphere can be generated using the gaze point location as the center and a preset target value as the radius. The region within the sphere is then considered the gaze point region. The set of vertices located within the sphere in the target 3D model can then be obtained. For ease of subsequent explanation, this vertex set will be defined as the target vertex set.

[0080] Understandably, in the model data of a 3D model, vertex buffers and index buffers are typically used to store the position, normals, and other data of each vertex. Therefore, in this embodiment, the position data of all vertices in the target vertex set can be obtained by querying the index from the index buffer and then querying the vertex position data from the vertex buffer using the index.

[0081] For example, the target 3D model includes several triangles. Assuming the first vertex of the i-th triangle falls into the sphere, the index of the first vertex of the i-th triangle, Index0, is: Index0 = IndexBuffer[3i], meaning that the 3i-th index is retrieved from the target triangle's index buffer to obtain Index0; based on Index0, the coordinates of the first vertex of the i-th triangle can be read from the vertex buffer of the target 3D model. Similarly, if the second vertex of the i-th triangle falls into the sphere, the index of the second vertex of the i-th triangle, Index1, is: Index1 = IndexBuffer[3i+1], meaning that the (3i+1)-th index is retrieved from the target triangle's index buffer to obtain Index1; based on Index1, the coordinates of the second vertex of the i-th triangle can be read from the vertex buffer of the target 3D model. If the third vertex of the i-th triangle falls into the sphere, then the index Index2 of the third vertex of the i-th triangle is: Index2 = IndexBuffer[3i+2], which means that the index (3i+2) is retrieved from the index buffer of the target triangle to obtain Index2; based on Index2, the coordinates of the third vertex of the i-th triangle can be read from the vertex buffer of the target 3D model.

[0082] Step S104: Perform vertex density analysis on the target vertex set to obtain the focal depth used for rendering the 3D scene.

[0083] In 3D rendering, the role of depth of focus includes at least the following: First, highlighting the subject: by adjusting the depth of focus, the main object is made clear while the background is blurred, thereby guiding the viewer's attention; Second, enhancing the sense of depth: by controlling the depth of focus, different objects in the 3D scene are made to present different degrees of clarity, enhancing the three-dimensionality and spatial sense of the image; Third, simulating realistic photography effects: by adjusting the depth of focus and depth of field parameters, the focusing and depth of field effects in real photography are simulated, making the rendered image more realistic.

[0084] To make the area being viewed by the user more clearly perceived by the rendering effect, this embodiment can perform vertex density analysis on the target vertex set to obtain the focus depth for rendering the 3D scene.

[0085] Next, this embodiment can render the 3D field based on the focal depth to obtain a rendering frame. Optionally, a hardware synchronization signal can be used to ensure that the timestamp of the eye-tracking data is precisely aligned with the rendering frame (specifically, to ensure that the timestamp of the last frame of eye-tracking data in the above multi-frame eye-tracking data is precisely aligned with the rendering frame) to compensate for processing delay.

[0086] The focal depth determination method provided in this application collects the user's eye movement data at a preset eye movement sampling rate during the user's viewing of a 3D scene, obtaining multiple frames of eye movement data within a target duration. Then, based on these multiple frames of eye movement data, the position of the user's gaze point within the target 3D model is determined. Considering that the vertex density around the gaze point characterizes the model complexity of the gaze point region, and that a more complex model requires a reduced focal depth to ensure that planes aligned with the gaze point remain sharp while those not aligned are less sharp, thus avoiding visual interference from misaligned planes and improving the user's ability to distinguish details at the gaze point, this application generates a sphere with the gaze point position as its center and a radius of a target value. It then obtains the target vertex set located within the sphere in the target 3D model, performs vertex density analysis on the target vertex set, and obtains the focal depth used for rendering the 3D scene. Therefore, this application can generate a focal depth that dynamically adjusts with the user's gaze, improving the user's ability to distinguish details at the gaze point and thus enhancing the user's viewing experience of the 3D scene.

[0087] In some embodiments of this application, the process of step S102, "determining the position of the user's gaze point in the target 3D model based on multiple frames of eye-tracking data," is described in detail.

[0088] Users may view multiple objects in a short period of time. To ensure the consistency and stability of the objects viewed by the user, this embodiment can optionally first determine whether the multi-frame eye-tracking data meets the preset effective gaze determination conditions. If so, the three-dimensional model that meets the effective gaze determination conditions is determined as the target three-dimensional model. Then, based on the multi-frame eye-tracking data, the position of the user's gaze point in the target three-dimensional model is determined.

[0089] Optionally, the effective gaze determination criteria include one or more of the following conditions: Condition 1 is that the total time the user gazes at a 3D model in the 3D scene within the target duration reaches a preset duration threshold; Condition 2 is that the gaze stability of the user gazing at a 3D model in the 3D scene within the target duration reaches a preset stability threshold.

[0090] Taking the valid gaze determination conditions including conditions 1 and 2 above as an example, the optional process of "determining the gaze point position of the user's gaze in the target 3D model based on multi-frame eye-tracking data" may include:

[0091] Multiple frames of eye-tracking data are divided into multiple groups according to the acquisition time sequence. Based on the eye-tracking data in each group, the gaze object corresponding to each group is determined to obtain the gaze objects corresponding to multiple groups respectively. Here, the gaze object refers to the object that the user gazes at in the three-dimensional scene.

[0092] Based on the gaze objects corresponding to multiple groups, determine whether there are target gaze objects whose gaze ratio reaches a preset ratio threshold. The gaze ratio of a gaze object is the ratio of the number of gazes of that gaze object to the total number of gazes.

[0093] If so, calculate the standard deviation of the user's gaze points based on multi-frame eye-tracking data;

[0094] If the standard deviation of the distribution is less than the preset standard deviation threshold, then the 3D model of the target object to be viewed is used as the target 3D model, and the model data of the target 3D model is obtained.

[0095] The fixation point location is determined based on multi-frame eye-tracking data and model data.

[0096] The aforementioned number of fixations and total number of fixations are obtained by counting the number of fixations on the objects corresponding to multiple groups. The preset ratio threshold is determined based on the preset time threshold, and the standard deviation threshold is obtained based on the display screen size of the display device of the 3D scene.

[0097] To enable those skilled in the art to better understand the above process, the following description provides an exemplary account.

[0098] Taking a target duration of 2 seconds and a first eye movement sampling rate of 120Hz as an example, the identification of the gaze object hit by the user's gaze can be performed every 100 milliseconds. That is, 12 frames of eye movement data are collected in the order of collection time (240 frames of eye movement data are collected in 2 seconds, so 12 frames of eye movement data are collected every 100 milliseconds as a group). Based on these 12 frames of eye movement data, three-dimensional ray detection is performed to obtain a gaze object (in this embodiment, the identity ID of the gaze object can be recorded); and so on, a total of 20 gaze object IDs are obtained from 240 frames of eye movement data.

[0099] Assume there are 20 gaze object IDs: gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 2, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, gaze object 1, and gaze object 2. Then, the gaze ratio of gaze object 1 = 17 (number of gazes of gaze object 1) / 20 (total number of gazes) = 0.85, the gaze ratio of gaze object 2 = 2 (number of gazes of gaze object 2) / 20 = 0.1, and the gaze ratio of gaze object 3 = 1 (number of gazes of gaze object 3) / 20 = 0.05.

[0100] Taking a preset ratio threshold of 0.8 (corresponding to a preset time threshold of 1600 milliseconds) as an example, since the gaze ratio of the gaze object 1 has reached the preset ratio threshold, it can be determined that 240 frames of eye-tracking data meet condition 1.

[0101] Next, based on the x-component in the normalized screen coordinates of the multi-frame eye-tracking data, the standard deviation of the user's gaze point in the x-direction is calculated; and based on the y-component in the normalized screen coordinates of the multi-frame eye-tracking data, the standard deviation of the user's gaze point in the y-direction is calculated. If both the standard deviations in the x-direction and y-direction are less than a preset standard deviation threshold (for example, the standard deviation threshold is 5% of the screen width), then the 240 frames of eye-tracking data are determined to satisfy condition 2.

[0102] Since the 240 frames of eye-tracking data simultaneously satisfy conditions 1 and 2, the 3D model of the object being gazed upon can be identified as the target 3D model. Next, the model data of the target 3D model is acquired, and the gaze point position is determined based on the 240 frames of eye-tracking data and the model data.

[0103] In one possible implementation, the process of "determining the fixation point location based on multi-frame eye-tracking data and model data" described above may include:

[0104] Each frame of eye-tracking data in the multi-frame eye-tracking data is processed into a ray equation for the user's gaze, so as to obtain multiple ray equations;

[0105] A hierarchical bounding box tree is constructed based on the model data, where several vertices of the target 3D model are connected to form several polygons, and each leaf node of the hierarchical bounding box tree represents a polygon.

[0106] For each ray equation in multiple ray equations: perform collision detection based on the ray equation and the hierarchical bounding box tree, obtain the leaf node in the hierarchical bounding box tree that intersects with the ray equation, and take the polygon represented by the leaf node as the polygon corresponding to the ray equation; thus obtaining the polygons corresponding to multiple ray equations respectively.

[0107] From the polygons corresponding to multiple ray equations, determine the polygon that is closest to the eye-tracking data acquisition device and use it as the target polygon;

[0108] Determine the coordinates of each vertex of the target polygon based on the model data;

[0109] Based on the coordinate data of each vertex of the target polygon, the coordinate data of the center point of the target polygon is determined as the position of the gaze point, wherein the center point is equidistant from each vertex.

[0110] Specifically, for each frame of eye-tracking data in a multi-frame eye-tracking dataset, this embodiment can obtain a ray equation in the following manner.

[0111] First, the normalized screen coordinates of the eye-tracking data frame are transformed to the standardized coordinate system defined by the graphics API (Application Programming Interface). The transformation formula is as follows (1) to map to a unified mathematical space for easy subsequent calculation.

[0112] Formula (1);

[0113] in, This represents the x-component of the normalized screen coordinates of the eye-tracking data for that frame. This represents the x-coordinate value of the normalized screen coordinates of the eye-tracking data frame transformed to the normalized coordinate system. This represents the y-component of the normalized screen coordinates of the eye-tracking data for that frame. This represents the y-coordinate value of the normalized screen coordinates of the eye-tracking data frame transformed to the normalized coordinate system. This represents the z-coordinate value of the eye-tracking data frame after the normalized screen coordinates have been transformed to the normalized coordinate system. =1 indicates a fixed orientation towards the far cutting surface.

[0114] Next, using the projection matrix and view matrix of the camera (i.e. the camera in the eye-tracking device), the coordinates in the normalized coordinate system are reversed to the three-dimensional world space to obtain the ray equation. The specific process can be referred to in formulas (2) to (4) below.

[0115] Formula (2);

[0116] Formula (3);

[0117] Formula (4);

[0118] in, This indicates the starting point of the ray, which is also the coordinate of the camera in three-dimensional world space; Represents the view matrix. Represents the projection matrix; This represents the direction vector of the ray, which is the direction of the target that the user's line of sight is pointing to; This represents a scalar parameter used to control the distance between a point on the ray and the starting point; Represents a point on a ray.

[0119] The calculations for multiple frames of eye-tracking data are performed according to the above formulas (1) to (4), which yields the ray equations corresponding to the multiple frames of eye-tracking data, i.e., multiple ray equations.

[0120] Next, a hierarchical bounding box tree (BVH) can be constructed based on the model data. Here, the hierarchical bounding box tree (BVH) is a tree-like data structure used to accelerate collision detection between rays and objects in a 3D scene. Its core idea is to decompose complex geometry into multiple nested bounding boxes (such as cuboids and spheres), and quickly eliminate regions that cannot intersect with rays by checking the intersection of bounding boxes layer by layer, thereby reducing the amount of computation.

[0121] Optionally, the process of constructing a hierarchical bounding box tree may include: determining the minimum axis-aligned bounding box of the target 3D model based on the model data; selecting the midpoint of the longest axis of the minimum axis-aligned bounding box; dividing the target 3D model into left and right parts to obtain left subset 1 composed of polygons in the left part and right subset 1 composed of polygons in the right part; then determining the minimum axis-aligned bounding box of left subset 1 based on the model data; selecting the midpoint of the longest axis of the minimum axis-aligned bounding box; dividing left subset 1 into left and right parts to obtain left subset 11 and right subset 12; similarly dividing right subset 1 to obtain left subset 21 and right subset 22; then dividing left subset 11, right subset 12, left subset 21, and right subset 22 according to the above process, until each polygon of the target 3D model corresponds to a child node (i.e., a leaf node).

[0122] Next, for each of the multiple ray equations, collision detection can be performed based on the ray equation and the hierarchical bounding box tree to obtain the leaf node in the hierarchical bounding box tree that intersects with the ray equation. The polygon represented by the leaf node is then used as the polygon corresponding to the ray equation.

[0123] Specifically, collision detection can be performed layer by layer according to the order of the hierarchical bounding box tree from the root node to the leaf node. If the ray equation intersects with the left subset 1, then all child nodes of the left subset 1 are recursively checked until the leaf node is reached. If the ray equation intersects with the right subset 1, then all child nodes of the right subset 1 are recursively checked until the leaf node is reached.

[0124] Taking the connection of several vertices of the target 3D model to form a triangle (a type of polygon) as an example, the following formula (5) can be used for collision detection between the ray equation and the leaf node.

[0125] Formula (5);

[0126] in, , and This represents the coordinates of the three vertices of the triangle represented by the leaf node. The process of obtaining these values ​​can be found in the following description. , and The value to be solved in formula (5) is, and satisfies .

[0127] here, , and The calculation process may include: first, following the method described in step S103 above, that is, obtaining the vertex coordinates in the vertex buffer by querying the index in the index buffer. , and The corresponding local coordinates are respectively , and Since the ray equation is an equation in three-dimensional world space, it is necessary to... , and The corresponding local coordinates are transformed to the world coordinate system to obtain the following results. , and The corresponding world coordinates , and ,Will As in the above formula (5) , As in the above formula (5) , As in the above formula (5) .

[0128] Among them, , and The process of transforming the local coordinates to the world coordinate system is as follows (6).

[0129] Formula (6);

[0130] in, It is a 4×4 matrix containing the position and orientation information of the target 3D model, and is a known matrix.

[0131] If the coordinates of the three vertices of the triangle corresponding to a certain leaf node are substituted into the above formula (5) and a solution is found, then the solution can be obtained. , and If the value of is determined, then the triangle represented by the leaf node is determined as the triangle corresponding to the ray equation.

[0132] By processing each ray equation according to the above process, we can obtain multiple polygons corresponding to each ray equation.

[0133] Since the human eye sees the polygon closest to the eye (or camera) when it gazes at the target 3D model, the polygon closest to the eye-tracking data acquisition device can be determined from the polygons corresponding to multiple ray equations and used as the target polygon.

[0134] Taking the polygon as a triangle as an example, as mentioned above, substituting each ray equation into formula (5) yields a set of... , and Multiple ray equations can yield multiple sets of results. , and This embodiment can determine that... The smallest ray equation corresponds to the triangle closest to the camera, which is also the target triangle.

[0135] Next, this embodiment can obtain the coordinate data of each vertex of the target polygon based on the model data of the target 3D model, such as the world coordinates mentioned above. , and .

[0136] Finally, based on the coordinates of each vertex of the target polygon, the coordinates of its center point can be determined as the fixation point location, where the center point is equidistant from each vertex. Here, the fixation point location is... ,in, Indicates the location of the fixation point. The x-component of the world coordinates of the gaze point. The y-component of the world coordinates of the gaze point. The z-component of the world coordinates of the gaze point.

[0137] In summary, the embodiments of this application can convert the user's line of sight into a ray equation from the camera to the gaze point position. Combined with collision detection with the hierarchical bounding box tree, the target polygon hit by the user's line of sight can be accurately identified. Then, the center point of the target polygon can be used as the gaze point position, which facilitates the subsequent calculation of the focal depth.

[0138] In some embodiments of this application, two modes can be set: a high-precision acquisition mode and a low-power monitoring mode. In the high-precision acquisition mode, eye movement data can be acquired at a higher eye movement sampling rate (i.e., the first eye movement sampling rate mentioned above), and the focal depth can be determined according to this application. In the low-power monitoring mode, eye movement data can be acquired at a lower eye movement sampling rate (i.e., the second eye movement sampling rate mentioned below), but the focal depth is not determined (only the gaze object ID is recorded).

[0139] Specifically, when the standard deviation of the distribution is less than the standard deviation threshold, while using the 3D model of the target gaze object as the target 3D model and acquiring the model data of the target 3D model, the system also collects the user's eye movement data at a preset first eye movement sampling rate to obtain multiple frames of eye movement data within the target duration. When the standard deviation of the distribution is greater than or equal to the standard deviation threshold, or when there is no target gaze object, the system does not use the 3D model of the target gaze object as the target 3D model and acquire the model data of the target 3D model, but continues to collect the user's eye movement data at a second eye movement sampling rate, where the second eye movement sampling rate is less than the first eye movement sampling rate.

[0140] In summary, this embodiment can execute in high-precision acquisition mode when it is found that the user has effectively gazed at the target object (i.e., both conditions 1 and 2 above are met). In this mode, the entire process is processed to obtain the focus depth. Experiments have verified that the CPU utilization rate in this mode is 30% and the GPU utilization rate is 40%. When it is found that the user has not effectively gazed at any object (i.e., conditions 1 and / or 2 above are not met), it can execute in low-power monitoring mode. In this mode, the focus depth is not calculated, only the gaze object ID is recorded, the CPU utilization rate is less than or equal to 5%, and the GPU is not activated, which reduces resource consumption and improves resource utilization.

[0141] In another possible implementation, considering the potential for rapid eye movements when acquiring multiple frames of eye-tracking data as described above, to avoid interference from rapid eye movements, the process of "acquiring the user's eye-tracking data according to a preset first eye-tracking sampling rate to obtain multiple frames of eye-tracking data within a target duration" in this embodiment may include: acquiring eye-tracking data according to the first eye-tracking sampling rate; after acquiring each frame of eye-tracking data, determining whether the frame displacement between the current frame and the previous frame is less than a preset frame displacement threshold; if so, controlling the timer to continue timing while continuing to acquire eye-tracking data until the timer's timing duration reaches the target duration, thus obtaining multiple frames of eye-tracking data; if not, controlling the timer to reset to 0 and continue timing while continuing to acquire eye-tracking data until the timer's timing duration reaches the target duration, thus obtaining multiple frames of eye-tracking data.

[0142] The process of collecting users' eye movement data according to the second eye movement sampling rate can be referred to the above-described process of collecting data according to the first eye movement sampling rate, and will not be repeated here.

[0143] In other words, during the eye-tracking data acquisition process, this embodiment can detect the frame displacement between two adjacent frames of eye-tracking data in real time. When the frame displacement is greater than or equal to the frame displacement threshold, it indicates that the user has experienced rapid eye movement. At this time, it is considered that the previously acquired eye-tracking data will lead to a decrease in the accuracy of the focal depth calculation. Therefore, the timer is reset, and the focal depth is calculated according to the newly acquired eye-tracking data after the reset, so as to improve the accuracy of the focal depth.

[0144] This embodiment eliminates rapid eye-tracking interference by using adjacent frame shifting, improving the accuracy of focus depth and thus enhancing the rendering effect of 3D scenes and the user's viewing experience.

[0145] In some other embodiments of this application, the process of step S104, "performing vertex density analysis on the target vertex set to obtain the focal depth for rendering the 3D scene," is described.

[0146] Considering that directly obtaining the focal depth from the gaze point position may result in the neglect of subtle undulations in high-density areas, and considering that there may be both high-density and sparse areas within the sphere, in order to achieve a better rendering effect for high-density areas to the human eye, this embodiment can optionally perform vertex density clustering on the target vertex set to obtain the target vertex cluster with the largest number of vertices, and then perform vertex density weighting processing based on the target vertex cluster to obtain the focal depth.

[0147] Optionally, the process of "performing vertex density clustering on the target vertex set to obtain the target vertex cluster with the most vertices" may include: determining the number of vertices in the target vertex set as the number of vertices inside the sphere; determining the density of the sphere based on the number of vertices inside the sphere; obtaining the clustering radius based on the density of the sphere, wherein the clustering radius and the density of the sphere are inversely correlated; clustering the target vertex set based on the clustering radius to obtain several vertex clusters; and determining the vertex cluster with the most vertices from the several vertex clusters as the target vertex cluster.

[0148] Optionally, the process of "determining the density of the target sphere based on the number of vertices inside the sphere" can be carried out using the following formula (7).

[0149] Formula (7);

[0150] in, This indicates the number of vertices inside the sphere. Indicates the target value. This represents the density of a sphere.

[0151] It should be noted that, in this embodiment, the target value The size can be determined based on the actual scenario. For example, if the overall accuracy of the scenario is high, the size can be reduced. To improve sensitivity, the aperture can be increased to 2mm. If the scene is rough, the aperture can be increased to 10mm to avoid missed detections.

[0152] Of course, the 2mm and 10mm mentioned above are merely examples and are not intended to limit this application.

[0153] Optionally, the cluster radius can be obtained from the density of the spheres using the following formula (8).

[0154] Formula (8);

[0155] in, Indicates the cluster radius; This represents the base radius, which is a preset value.

[0156] As can be seen from formula (8), the higher the density of the fixation point region, the smaller the cluster radius, which allows for better focus on micro-details; the lower the density of the fixation point region, the larger the cluster radius, which allows for better coverage of macro-structures.

[0157] Optionally, this embodiment may employ the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm based on cluster radius. Cluster the target vertex set to obtain several vertex clusters, and then determine the vertex cluster with the most vertices from the several vertex clusters as the target vertex cluster.

[0158] Optionally, the process of "weighting vertex density based on the target vertex cluster to obtain the focal depth" may include: generating the minimum bounding box of the target vertex cluster and determining the volume of the minimum bounding box; determining the number of vertices contained in the target vertex cluster; determining the local density based on the number of vertices contained in the target vertex cluster and the volume of the minimum bounding box; obtaining the depth value of each vertex in the target vertex cluster from the position data of each vertex in the target vertex cluster; and calculating the average value by weighting the local density and the depth value of each vertex in the target vertex cluster to obtain the focal depth.

[0159] Here, the minimum bounding box of the target vertex cluster can be a spherical bounding box, a cuboid bounding box, or any other bounding box. This embodiment does not impose any specific limitations.

[0160] Optionally, in this embodiment, the number of vertices contained in the target vertex cluster can be divided by the volume of the minimum bounding box, and the quotient obtained can be used as the local density. .

[0161] In this embodiment, the position data of each vertex in the target vertex cluster can be referred to the preceding text. , and The calculation process will not be elaborated here.

[0162] Since the vertices in the target vertex cluster may be at different depths and scales, in order to unify the analysis, this embodiment can convert the position data of each vertex in the target vertex cluster to a local coordinate system with the gaze point position as the origin.

[0163] Specifically, taking the j-th vertex in the target vertex cluster as an example, the position data of the j-th vertex is used... If the position data of the j-th vertex is converted to the local coordinate system, the process is as follows (9).

[0164] Formula (9);

[0165] in, This represents the coordinates of the j-th vertex in the target vertex cluster in the local coordinate system. This represents the coordinates of the j-th vertex in the target vertex set in the world coordinate system.

[0166] Then, for the j-th vertex in the target vertex cluster, the coordinates of the j-th vertex in the local coordinate system are... The Z component is extracted and used as the depth value of the j-th vertex. Similarly, the depth values ​​of other vertices can be obtained.

[0167] Optionally, the process of “taking the weighted average of the local density and the depth values ​​of all vertices in the target vertex cluster to obtain the focal depth” can be carried out using the following formula (10).

[0168] Formula (10);

[0169] in, Indicates the depth of focus. This indicates the number of vertices contained in the target vertex cluster.

[0170] Compared to the traditional DBSCAN clustering algorithm which uses a fixed radius, this embodiment provides a clustering radius based on vertex density, which can better distinguish the vertex density differences in different regions of the target 3D model. As a result, the focal depth obtained is closer to the detail distribution, thus enabling high-density areas to present a better rendering effect to the human eye.

[0171] In some other embodiments of this application, after obtaining the focal depth, the depth of field parameter value can be determined based on the focal depth, and the three-dimensional scene can be rendered based on the depth of field parameter value and the focal depth.

[0172] Optionally, the process of determining the depth-of-field parameter value based on the focal depth may include: for each pixel in the 3D scene, determining the depth difference corresponding to the pixel based on the pixel depth and the focal depth, so as to obtain the depth difference corresponding to all pixels in the 3D scene; dividing the 3D scene into multiple rendering regions based on the depth differences corresponding to all pixels in the 3D scene, wherein different rendering regions use different rendering techniques; and determining the depth-of-field parameter value of the pixel based on the depth difference corresponding to each pixel in the 3D scene and the rendering region in which the pixel is located, so as to obtain the depth-of-field parameter value corresponding to all pixels in the 3D scene.

[0173] Optionally, the process of "determining the depth difference corresponding to the pixel based on the pixel depth and focus depth" can be carried out using the following formula (11).

[0174] Formula (11);

[0175] in, Represents the pixel depth of a pixel in a 3D scene. This represents the depth difference corresponding to pixels in a 3D scene.

[0176] Optionally, for each pixel in the 3D scene, when the pixel satisfies When the distance is ≤1cm, the pixel is assigned to the focal core region; when the distance is ≤1cm, the pixel is assigned to the focal core region. When the pixel is ≤5cm, it is classified into the transition region. When the pixel meets the following conditions... When the pixel length is greater than 5cm, it is classified as an edge region.

[0177] Optionally, for the focal core area, high-precision PBR (Physically Based Rendering) materials and real-time ray tracing (RTX) rendering technology can be used, accounting for 60% of the performance; for the transition area, medium-precision PBR materials and screen space reflection (SSR) rendering technology can be used, accounting for 30% of the performance; for the edge area, low-poly model, pre-baked lighting and 2D parallax mapping rendering technology can be used, accounting for 10% of the performance.

[0178] Optionally, the foreground blur radius and background blur radius of each pixel in the 3D scene can be calculated according to the following formulas (12) and (13).

[0179] Formula (12);

[0180] Formula (13);

[0181] in, Represents the foreground blur radius of a pixel in a 3D scene. Represents the background blur radius of a pixel in a 3D scene. and Art control coefficient (default) ).

[0182] Optionally, for each pixel in the 3D scene, if the rendering area where the pixel is located is the focus core area, then the blur kernel R of the pixel is 0 (i.e., no blur). If the rendering area where the pixel is located is the transition area, then the blur kernel R of the pixel is any value in the range of 3-7 (the specific value can be determined according to the actual scene). If the rendering area where the pixel is located is the edge area, then the blur kernel R of the pixel is any value in the range of 10-15 (the specific value can be determined according to the actual scene).

[0183] Therefore, the foreground blur radius, background blur radius, and blur kernel mentioned above are the depth-of-field parameter values ​​in this embodiment.

[0184] In summary, this embodiment can not only generate a focus depth that dynamically adjusts with the user's gaze, but also generate a depth-of-field parameter value that dynamically adjusts with the user's gaze. Based on this focus depth and depth-of-field parameter value that dynamically adjust with the user's gaze, the rendered 3D scene can have better detail perception in the high-density area of ​​the user's gaze point, thereby improving the rendering effect of the 3D scene and the user's viewing experience.

[0185] The above describes a method for determining focal depth provided by an embodiment of this application. The following describes an apparatus for performing the above-described method for determining focal depth.

[0186] Please see Figure 2 , Figure 2 This is a schematic diagram of a focal depth determining device provided in an embodiment of this application. Figure 2 As shown, the focal depth determining device may include:

[0187] The eye-tracking data acquisition module 201 is used to collect the user's eye-tracking data according to a preset first eye-tracking sampling rate during the user's viewing of a 3D scene, so as to obtain multiple frames of eye-tracking data collected within a target duration.

[0188] The gaze point location determination module 202 is used to determine the gaze point location of the user's gaze in the target 3D model based on multi-frame eye-tracking data. The target 3D model refers to the 3D model that the user gazes at in the 3D scene, and the target 3D model includes several vertices.

[0189] The vertex determination module 203 inside the sphere is used to generate a sphere with the sphere center as the foveation point position and the radius as the target value, and to obtain the set of vertices located inside the sphere in the target 3D model as the target vertex set, wherein the target vertex set includes the position data of each of the multiple vertices;

[0190] The focus depth determination module 204 is used to perform vertex density analysis on the target vertex set to obtain the focus depth for rendering the 3D scene.

[0191] In one possible implementation, the gaze point determination module described above, when determining the gaze point position of the user's gaze in the target 3D model based on multi-frame eye-tracking data, can specifically be used for:

[0192] Multiple frames of eye-tracking data are divided into multiple groups according to the acquisition time sequence. Based on the eye-tracking data in each group, the gaze object corresponding to each group is determined, so as to obtain the gaze objects corresponding to multiple groups respectively.

[0193] Based on the gaze objects corresponding to multiple groups, determine whether there are target gaze objects whose gaze ratio reaches a preset ratio threshold. The gaze ratio of a gaze object is the ratio of the number of gazes of that gaze object to the total number of gazes.

[0194] If so, calculate the standard deviation of the user's gaze points based on multi-frame eye-tracking data;

[0195] If the standard deviation of the distribution is less than the preset standard deviation threshold, then the 3D model of the target object to be viewed is used as the target 3D model, and the model data of the target 3D model is obtained.

[0196] The fixation point location is determined based on multi-frame eye-tracking data and model data.

[0197] In one possible implementation, the gaze point location determination module described above, when determining the gaze point location based on multi-frame eye-tracking data and model data, can specifically be used for:

[0198] Each frame of eye-tracking data in the multi-frame eye-tracking data is processed into a ray equation for the user's gaze, so as to obtain multiple ray equations;

[0199] A hierarchical bounding box tree is constructed based on the model data, where several vertices of the target 3D model are connected to form several polygons, and each leaf node of the hierarchical bounding box tree represents a polygon.

[0200] For each ray equation in multiple ray equations: perform collision detection based on the ray equation and the hierarchical bounding box tree, obtain the leaf node in the hierarchical bounding box tree that intersects with the ray equation, and take the polygon represented by the leaf node as the polygon corresponding to the ray equation; thus obtaining the polygons corresponding to multiple ray equations respectively.

[0201] From the polygons corresponding to multiple ray equations, determine the polygon that is closest to the eye-tracking data acquisition device and use it as the target polygon;

[0202] Determine the coordinates of each vertex of the target polygon based on the model data;

[0203] Based on the coordinate data of each vertex of the target polygon, the coordinate data of the center point of the target polygon is determined as the position of the gaze point, wherein the center point is equidistant from each vertex.

[0204] In one possible implementation, the focus depth determining device provided in this application may further include: a mode adjustment module.

[0205] The mode adjustment module is used to, when the standard deviation of the distribution is less than the standard deviation threshold, simultaneously acquire the model data of the target 3D model by using the 3D model of the target gaze object as the target 3D model, and return to collect the user's eye movement data at a preset first eye movement sampling rate to obtain multiple frames of eye movement data within the target duration; when the standard deviation of the distribution is greater than or equal to the standard deviation threshold, or when there is no target gaze object, it does not acquire the model data of the target 3D model by using the 3D model of the target gaze object as the target 3D model, and continues to collect the user's eye movement data at a second eye movement sampling rate, wherein the second eye movement sampling rate is less than the first eye movement sampling rate.

[0206] In one possible implementation, the aforementioned focus depth determination module, when performing vertex density analysis on the target vertex set to obtain the focus depth for rendering the 3D scene, can specifically be used for:

[0207] Perform vertex density clustering on the target vertex set to obtain the target vertex cluster with the largest number of vertices;

[0208] The focal depth is obtained by performing vertex density weighting on the target vertex cluster.

[0209] In one possible implementation, the aforementioned focus depth determination module, when performing vertex density clustering on the target vertex set to obtain the target vertex cluster with the largest number of vertices, can specifically be used for:

[0210] Determine the number of vertices in the target vertex set, and use that number as the number of vertices inside the sphere;

[0211] The density of a sphere is determined by the number of vertices within it.

[0212] The cluster radius is obtained from the density of the spheres, and the cluster radius is inversely correlated with the density of the spheres.

[0213] Clustering of the target vertex set based on the clustering radius yields several vertex clusters;

[0214] From a number of vertex clusters, determine the vertex cluster with the largest number of vertices and use it as the target vertex cluster.

[0215] In one possible implementation, the focus depth determination module described above, when performing vertex density weighting processing based on the target vertex cluster to obtain the focus depth, can specifically be used for:

[0216] Generate the minimum bounding box of the target vertex cluster and determine the volume of the minimum bounding box;

[0217] Determine the number of vertices contained in the target vertex cluster;

[0218] The local density is determined based on the number of vertices contained in the target vertex cluster and the volume of the minimum bounding box;

[0219] Obtain the depth value of each vertex in the target vertex cluster from the position data of each vertex in the target vertex cluster;

[0220] The focal depth is obtained by taking the weighted average of the local density and the depth values ​​of all vertices in the target vertex cluster.

[0221] Each module in the aforementioned depth-of-focus determination device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0222] This application also provides an electronic device, which may include at least one processor and a memory connected to the processor, wherein:

[0223] Memory is used to store computer programs;

[0224] The processor is used to execute computer programs to enable the electronic device to implement any of the focus depth determination methods provided in the embodiments of this application.

[0225] refer to Figure 3 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0226] like Figure 3 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. When the electronic device is powered on, the RAM 703 also stores various programs and data required for the operation of the electronic device. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0227] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, memory cards, hard drives, etc.; and communication devices 709. Communication device 709 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0228] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the focus depth determination methods provided in this application.

[0229] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the focus depth determination methods provided in this application.

[0230] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0231] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0232] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0233] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method of focus depth determination, characterized by, Comprise: In the process of a user watching a three-dimensional scene, eye movement data of the user is collected at a preset first eye movement sampling rate to obtain a plurality of frames of eye movement data collected within a target time length; According to the plurality of frames of eye movement data, a gaze point position of a user's line of sight in a target three-dimensional model is determined, wherein the target three-dimensional model refers to a three-dimensional model at which the user gazes in the three-dimensional scene, and the target three-dimensional model includes a plurality of vertices; A sphere with a target value as a radius is generated with the gaze point position as a sphere center, a vertex set in the target three-dimensional model located in the sphere is obtained as a target vertex set, wherein the target vertex set includes position data of each of a plurality of vertices; A vertex density analysis is performed on the target vertex set to obtain a focal depth for rendering the three-dimensional scene; wherein the target vertex set is subjected to vertex density clustering processing to obtain a target vertex cluster with the most vertices; a local density is determined according to the number of vertices included in the target vertex cluster and the volume of a minimum bounding box; a depth value of each of all vertices in the target vertex cluster is obtained from the position data of each of all vertices in the target vertex cluster; and a focal depth is obtained by weighted averaging according to the local density and the depth value of each of all vertices in the target vertex cluster.

2. The focal depth determination method of claim 1, wherein, The determination of the gaze point position of the user's line of sight in the target three-dimensional model according to the plurality of frames of eye movement data comprises: The plurality of frames of eye movement data are evenly divided into a plurality of groups in chronological order, and a gaze object corresponding to each group is determined according to the eye movement data in each group to obtain gaze objects corresponding to the plurality of groups respectively; It is determined whether there is a target gaze object with a gaze ratio reaching a preset ratio threshold according to the gaze objects corresponding to the plurality of groups respectively, wherein the gaze ratio of a gaze object is a ratio of a number of gazes at the gaze object to a total number of gazes; If so, a distribution standard deviation of the gaze point of the user is calculated according to the plurality of frames of eye movement data; If the distribution standard deviation is less than a preset standard deviation threshold, a three-dimensional model of the target gaze object is taken as the target three-dimensional model, and model data of the target three-dimensional model is obtained; The gaze point position is determined according to the plurality of frames of eye movement data and the model data.

3. The focal depth determination method of claim 2, wherein, The determination of the gaze point position according to the plurality of frames of eye movement data and the model data comprises: Each frame of eye movement data in the plurality of frames of eye movement data is processed into a ray equation of a user's line of sight to obtain a plurality of ray equations; A hierarchical bounding box tree is constructed according to the model data, wherein a plurality of vertices of the target three-dimensional model form a plurality of polygons by connection, and each leaf node of the hierarchical bounding box tree represents one of the polygons; For each ray equation in the plurality of ray equations: collision detection is performed according to the ray equation and the hierarchical bounding box tree to obtain a leaf node of the hierarchical bounding box tree intersecting the ray equation, and a polygon represented by the leaf node is taken as a polygon corresponding to the ray equation; and the polygons corresponding to the plurality of ray equations are obtained respectively. determine a polygon closest to the collection device of the eye movement data from the plurality of polygons corresponding to the plurality of ray equations respectively as a target polygon; determine coordinate data of each vertex of the target polygon according to the model data; determine coordinate data of a center point of the target polygon as the gaze point position according to the coordinate data of each vertex of the target polygon, wherein the center point is respectively equal in distance to each vertex.

4. The focal depth determination method of claim 2, wherein, Further comprising: in the case where the distribution standard deviation is less than the standard deviation threshold, return the collection of the eye movement data of the user according to the preset first eye movement sampling rate to obtain a plurality of frames of eye movement data collected within a target time length while performing the obtaining of the model data of the target three-dimensional model of the target gaze object as the target three-dimensional model; in the case where the distribution standard deviation is greater than or equal to the standard deviation threshold or there is no target gaze object, do not perform the obtaining of the model data of the target three-dimensional model of the target gaze object as the target three-dimensional model, and continue to collect the eye movement data of the user according to a second eye movement sampling rate, wherein the second eye movement sampling rate is less than the first eye movement sampling rate.

5. The focal depth determination method of claim 1, wherein, The vertex density clustering processing on the target vertex set to obtain a target vertex cluster with the largest number of vertices includes: determining the number of vertices in the target vertex set as the number of in-sphere vertices; determining the density of the sphere according to the number of in-sphere vertices; obtaining a clustering radius according to the density of the sphere, wherein the clustering radius and the density of the sphere are inversely related; clustering the target vertex set based on the clustering radius to obtain a plurality of vertex clusters; determining a vertex cluster with the largest number of vertices from the plurality of vertex clusters as the target vertex cluster.

6. The focal depth determination method of claim 1, wherein, The determination of the local density according to the number of vertices contained in the target vertex cluster and the volume of the minimum bounding box includes: generating a minimum bounding box of the target vertex cluster to determine the volume of the minimum bounding box; determining the number of vertices contained in the target vertex cluster; determining the local density according to the number of vertices contained in the target vertex cluster and the volume of the minimum bounding box.

7. A focus depth determination apparatus, characterized by Comprising: an eye movement data acquisition module, configured to collect eye movement data of a user according to a preset first eye movement sampling rate during viewing of a three-dimensional scene by the user to obtain a plurality of frames of eye movement data collected within a target time length; a gaze point position determination module, configured to determine a gaze point position of a user's line of sight in a target three-dimensional model according to the plurality of frames of eye movement data, wherein the target three-dimensional model refers to a three-dimensional model gazed at by the user in the three-dimensional scene, and the target three-dimensional model includes a plurality of vertices; an in-sphere vertex determination module, configured to generate a sphere with a target value as a radius with the gaze point position as a sphere center, acquire a vertex set located in the sphere in the target three-dimensional model as a target vertex set, and include position data of each of a plurality of vertices in the target vertex set; A focus depth determination module is configured to perform vertex density analysis on the target vertex set to obtain a focus depth for rendering the three-dimensional scene; wherein the target vertex set is subjected to vertex density clustering processing to obtain a target vertex cluster with the largest number of vertices; a local density is determined according to the number of vertices contained in the target vertex cluster and the volume of the minimum bounding box; a depth value of each vertex in the target vertex cluster is obtained from position data of each vertex in the target vertex cluster; and the focus depth is obtained by performing weighted average according to the local density and the depth value of each vertex in the target vertex cluster.

8. An electronic device, comprising: An electronic device includes at least one processor and a memory connected to the processor, wherein: The memory is configured to store a computer program; The processor is configured to execute the computer program to enable the electronic device to implement the focus depth determination method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the focus depth determination method according to any one of claims 1 to 6.

Images