Track acquisition method, device and system

Abstract

The invention provides a track acquisition method, device and system. The method comprises the steps of obtaining a three-dimensional space model of a space environment where the wearable device is located; then, acquiring an eye sight line direction vector when the user wears the equipment; and then, based on the sight line direction vector and the three-dimensional space model, determining a fixation point position of the user in the space, and generating a motion track of the user in the space. The movement path, the speed change, the head orientation and rotation and the like of the user can be analyzed through the movement track, so that more personalized and interactive experience is provided for deep analysis of the watching behavior of the user, and a new angle is provided for exploring visual behaviors.

Description

Technical Field

[0001] The present disclosure relates to the field of display technology, and in particular to a trajectory acquisition method, device and system. Background Art

[0002] With the rapid development of augmented reality (AR) and virtual reality (VR) technologies, they are increasingly being used in education, entertainment, exhibitions, commerce, etc. Taking museums and exhibition halls as an example, AR devices or VR devices can provide an attractive and personalized visiting experience.

[0003] However, existing devices are limited in functionality and usually provide general services, such as explanations, but are unable to provide users with personalized guided tour services, which is not conducive to improving user satisfaction. Summary of the invention

[0004] The present disclosure provides a trajectory acquisition method, device and system to solve the above technical problems.

[0005] According to a first aspect of the present disclosure, a trajectory acquisition method is provided, which is applicable to a wearable device worn by a user, and the method comprises:

[0006] Obtaining a three-dimensional space model of the space environment in which the wearable device is located, and obtaining a sight line direction vector of the eyes of the user wearing the wearable device;

[0007] Determining a gaze point of a user in the three-dimensional space model according to the eye direction vector and the three-dimensional space model;

[0008] A movement trajectory of the user is generated according to all gaze points; the movement trajectory includes a motion trajectory and a gaze point trajectory.

[0009] Optionally, obtaining a three-dimensional space model of the space environment in which the wearable device is located includes:

[0010] Acquire depth data, RGB image data, and point cloud data collected by the wearable device;

[0011] Aligning the spatial points in the depth data with the pixel points in the RGB image data to obtain an initial spatial model;

[0012] Acquire the polygons of each surface and the normal vectors of each space point in the initial space model according to the point cloud data;

[0013] Perform normal mapping on the initial space model based on the normal vector to obtain a three-dimensional space model.

[0014] Optionally, obtaining a sight line direction vector of an eye of a user wearing the wearable device includes:

[0015] Obtaining a pupil position and a reflection point position according to an eye image of the user's eye;

[0016] The sight line direction vector is determined based on a preset eyeball model, the pupil position and the reflection point position.

[0017] Optionally, determining a gaze point of the user in the three-dimensional space model according to the eye direction vector and the three-dimensional space model includes:

[0018] Get the head space position and rotation angle;

[0019] Converting the sight direction vector to the three-dimensional space model according to a preset rotation matrix;

[0020] Acquire the viewpoint position according to the head space position, the rotation matrix and the preset offset;

[0021] Determine at least one intersection point between the eye ray and the three-dimensional space model, and use the nearest intersection point as the gaze point; or, when it is determined that the eye ray does not intersect with the three-dimensional space model, obtain an invalid gaze point.

[0022] Optionally, generating a movement trajectory of the user according to all gaze points includes:

[0023] Acquire a region of interest in the three-dimensional space model;

[0024] Determine the region of interest where the gaze point is located, and obtain a gaze event;

[0025] Get statistics of fixation events for each region of interest;

[0026] A focus point trajectory matching the user's motion trajectory is generated according to the statistical data to obtain the movement trajectory.

[0027] Optionally, obtain statistics on gaze events for each region of interest, including:

[0028] Get the event distance from the fixation point in each fixation event to the center point of the region of interest;

[0029] Determine an event threshold of the gaze event according to a time frame where the gaze event is located;

[0030] When the event distance is less than or equal to the event threshold, determining that the region of interest is in a focused state within the time frame;

[0031] The duration of the focusing state of the region of interest is obtained as the first statistical data of the gaze event of the region of interest.

[0032] Optionally, determining an event threshold of the gaze event according to a time frame in which the gaze event occurs includes:

[0033] Get the distance from the viewpoint to the gaze point, and get the eye viewing angle;

[0034] Obtaining a tangent value of half the viewing angle; the tangent value represents a calculated distance between the viewpoint and the gaze point;

[0035] The product of the distance and the tangent value is obtained, and twice the product is used as the event threshold.

[0036] Optionally, the method further comprises:

[0037] Get the cumulative duration of all the regions of interest in the preset area;

[0038] Get the total area of ​​all regions of interest in the preset area;

[0039] A ratio of the accumulated duration to the total area of ​​the region is obtained as second statistical data of gaze events in the region of interest.

[0040] According to a second aspect of the present disclosure, a trajectory acquisition device is provided, which is applicable to a wearable device worn by a user, and the device comprises:

[0041] A spatial model acquisition module is used to acquire a three-dimensional spatial model of the spatial environment in which the wearable device is located;

[0042] A direction vector acquisition module, used to acquire a sight line direction vector of the eye of the user wearing the wearable device;

[0043] A gaze point acquisition module, used for determining a gaze point of a user in the three-dimensional space model according to the eye direction vector and the three-dimensional space model;

[0044] The movement trajectory acquisition module is used to generate the movement trajectory of the user according to all the gaze points; the movement trajectory includes a motion trajectory and a gaze point trajectory.

[0045] Optionally, the space model acquisition module includes:

[0046] A data acquisition submodule, used to acquire depth data, RGB image data and point cloud data collected by the wearable device;

[0047] An initial model acquisition submodule, used to align the spatial points in the depth data with the pixel points in the RGB image data to obtain an initial spatial model;

[0048] A normal vector acquisition submodule, used for acquiring the normal vectors of the polygons of each surface and each space point in the initial space model according to the point cloud data;

[0049] The space model acquisition submodule is used to perform normal mapping on the initial space model based on the normal vector to obtain a three-dimensional space model.

[0050] Optionally, the direction vector acquisition module includes:

[0051] A position acquisition submodule, used to acquire the pupil position and the reflection point position according to the eye image of the user's eye;

[0052] The vector acquisition submodule is used to determine the sight line direction vector based on a preset eyeball model, the pupil position and the reflection point position.

[0053] Optionally, the gaze point acquisition module includes:

[0054] The position angle acquisition submodule is used to obtain the head spatial position and rotation angle;

[0055] A direction conversion submodule, used for converting the sight direction vector into the three-dimensional space model according to a preset rotation matrix;

[0056] A viewpoint position acquisition submodule, used to acquire the viewpoint position according to the head space position, the rotation matrix and a preset offset;

[0057] The gaze point acquisition submodule is used to determine at least one intersection point between the eye ray and the three-dimensional space model, and use the nearest intersection point as the gaze point; or, when it is determined that the eye ray does not intersect with the three-dimensional space model, obtain an invalid gaze point.

[0058] Optionally, the movement trajectory acquisition module includes:

[0059] An area of ​​interest acquisition submodule is used to acquire an area of ​​interest in the three-dimensional space model;

[0060] A gaze event determination submodule, used to determine the region of interest where the gaze point is located, and obtain a gaze event;

[0061] A statistical data acquisition submodule is used to obtain statistical data of gaze events in each region of interest;

[0062] The movement trajectory acquisition submodule is used to generate a focus point trajectory matching the user movement trajectory according to the statistical data to obtain the movement trajectory.

[0063] Optionally, the statistical data acquisition submodule includes:

[0064] An event distance acquisition unit, used for acquiring the event distance from the gaze point in each gaze event to the center point of the region of interest mj;

[0065] An event threshold acquisition unit, used for determining an event threshold of the gaze event according to a time frame where the gaze event is located;

[0066] A focusing state determining unit, configured to determine that the region of interest is in a focusing state within the time frame when the event distance is less than or equal to the event threshold;

[0067] The first statistical data acquisition unit is used to acquire a duration during which the region of interest is in a focused state as first statistical data of a gaze event of the region of interest.

[0068] Optionally, the event threshold acquisition unit includes:

[0069] A distance acquisition subunit, used to acquire the distance from the viewpoint to the gaze point;

[0070] An eye perspective acquisition subunit, used to acquire the eye perspective;

[0071] A tangent value acquisition subunit, used to acquire a tangent value of half the viewing angle; the tangent value represents a calculated distance between the viewpoint and the gaze point;

[0072] The event threshold acquisition subunit is used to acquire the product of the distance and the tangent value, and take twice the product as the event threshold.

[0073] Optionally, the statistical data acquisition submodule further includes:

[0074] An accumulated duration acquisition unit, used to acquire the accumulated duration of all regions of interest in a preset area;

[0075] A total area acquisition unit, used to acquire the total area of ​​all regions of interest in a preset area;

[0076] The second statistical data acquisition unit is used to acquire the ratio of the accumulated duration to the total area of ​​the region as the second statistical data of the gaze event of the region of interest.

[0077] According to a third aspect of the present disclosure, a trajectory acquisition system is provided, comprising:

[0078] A processor and a memory; the memory is used to store a computer program executable by the processor;

[0079] The processor is used to execute the computer program in the memory to implement the method as described in any one of the first aspects.

[0080] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0081] The trajectory acquisition method provided in this embodiment includes: obtaining a three-dimensional space model of the space environment in which the wearable device is located, and obtaining the sight direction vector of the eyes of the user wearing the wearable device; determining the user's gaze point in the three-dimensional space model according to the eye direction vector and the three-dimensional space model; generating the user's movement trajectory according to all the gaze points; the movement trajectory includes a motion trajectory and a gaze point trajectory. In this way, this embodiment can obtain the user's gaze point trajectory and analyze the user's gaze behavior at different locations, which is conducive to continuously providing users with relevant content of the focus area and improving user satisfaction; further, it can optimize the content of the three-dimensional space layout and improve the user's subsequent viewing satisfaction.

[0082] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 The present invention is a flow chart of a trajectory acquisition method according to an embodiment of the present invention.

[0084] Figure 2 The present invention is a flowchart of obtaining a gaze point according to an embodiment of the present invention.

[0085] Figure 3 The present invention is a flowchart of obtaining a movement trajectory according to an embodiment of the present invention.

[0086] Figure 4 The present invention is a flowchart of obtaining first statistical data according to an embodiment of the present invention.

[0087] Figure 5 The present invention is a flowchart of a visual movement trajectory according to an embodiment of the present invention.

[0088] Figure 6 This is a simulation diagram of a visualized movement trajectory according to an embodiment of the present disclosure.

[0089] Figure 7 This is a simulation diagram of a visualized movement trajectory according to an embodiment of the present disclosure.

[0090] Figure 8 This is a simulation diagram of statistical data of an area of ​​interest being gazed at by a certain position in a moving trajectory according to an embodiment of the present disclosure.

[0091] Fig. 9 The present invention is a block diagram of a trajectory acquisition device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0092] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.

[0093] The embodiments of the present disclosure provide a trajectory acquisition method, device and system. The trajectory acquisition method can be applied to wearable devices. Wearable devices can include virtual reality devices, augmented reality devices or mixed reality devices, etc., which can be selected according to specific scenarios.

[0094] In one embodiment, see Figure 1 , the trajectory acquisition method may include steps 11 to 14.

[0095] In step 11, a three-dimensional space model of the space environment in which the wearable device is located is obtained, and a sight line direction vector of the eyes of the user wearing the wearable device is obtained.

[0096] In one embodiment, when a user wears a wearable device to visit a space environment such as a museum, memorial hall or science and technology museum, it is necessary to establish a mapping relationship between the user and the space environment. To this end, in one embodiment, the sight direction vector of the user wearing the wearable device and the three-dimensional space model of the space environment where the wearable device is located can be obtained.

[0097] Taking obtaining the sight direction vector of the eye of the user wearing the wearable device as an example, the wearable device may include an infrared camera or a camera array to obtain the user's eye image.

[0098] Taking an infrared camera as an example, an infrared camera can use an infrared light source to generate infrared light to illuminate the user's eyes, and the infrared camera can capture the reflected light to obtain a high-contrast eye image. It is understandable that infrared light is invisible to the user and does not affect the user's visual experience. Taking a micro camera array as an example, a micro camera array can collect a user's eye image.

[0099] In one embodiment, the eye image may be preprocessed. For example, the eye image is converted into a grayscale image to simplify subsequent processing. Then, the grayscale image is subjected to histogram equalization or logarithmic transformation to improve the contrast of the eye features. After that, the eye image is filtered to remove noise in the image. In addition, a face detection algorithm (such as HAAR features, CNN model, etc.) is used to locate the facial area, and the eye area is located in the facial area.

[0100] After obtaining the eye region in the eye image, edge detection can be performed on the eye region, such as Canny edge detection algorithms, to extract the edge of the pupil; then, ellipse fitting is performed on the edge points to determine the pupil shape and pupil (center) position. Since the corneal reflection point of the eye appears as a highlight point in the eye, it is extracted using threshold segmentation, and the center of mass of the highlight point is calculated to obtain the precise position.

[0101] Assuming that the eyeball is spherical, a geometric model of the eyeball can be established to obtain an eyeball model. It is understandable that since the position of the light source is known, the position of the corneal reflection point depends on the rotation angle of the eyeball. The sight direction vector is calculated based on the above eyeball model, the pupil (center) position and the corneal reflection point position, as shown in formula (1).

[0102] G = f(Pp, Pr) (1)

[0103] In formula (1), Pp represents the pupil position, Pr represents the corneal reflection point, and G represents the sight direction vector.

[0104] Taking the acquisition of a three-dimensional spatial model of the spatial environment in which the wearable device is located as an example, the wearable device may include a depth sensor, such as a time-of-flight sensor or a structured light sensor, which can acquire a depth image of the spatial environment in which the user is located and simultaneously acquire an RGB image. It is understandable that the depth image matches the RGB image.

[0105] In this example, the above-mentioned depth image can be preprocessed. For example, the depth image is subjected to median filtering to remove isolated noise points in the depth image to reduce the impact of random noise. The depth image is then subjected to bilateral filtering to smooth the depth image while retaining edge details, thereby improving the quality of the depth data. Afterwards, the depth image can be subjected to nonlinear correction to correct the nonlinear distortion caused by the measurement error of the sensor. Furthermore, the depth image can be subjected to distortion correction to improve the image distortion caused by lens distortion and determine the accuracy of the depth data.

[0106] Considering that the depth sensor and RGB camera of the wearable device have been pre-calibrated, the depth image can be aligned with the RGB image to determine that the same pixel position corresponds to the same spatial point. Then, using the depth value and the camera intrinsic parameters (focal length, spatial point coordinates), each pixel in the depth image is converted to a three-dimensional coordinate in the camera coordinate system. The pixels in the camera coordinate system are then converted to the world coordinate system, as shown in equation (2).

[0107] Pworld=R*Pcamera+T1 (2)

[0108] In formula (2), T1 represents the translation vector, R represents the rotation matrix, Pcamera represents the pixel in the camera coordinate system, and Pworld represents the spatial point in the world coordinate system.

[0109] Considering that the depth image and the RGB image have been aligned, the color value can be obtained from the corresponding RGB image and annotated to each spatial point to obtain a color point cloud, that is, to obtain the initial spatial model.

[0110] After obtaining the initial spatial model, the space can be divided into cubic units of preset size, also called voxels. Then, the point cloud density in each voxel is counted. For example, after traversing each voxel, the shape of the isosurface is determined according to the value of the voxel corner point, and then the preset topology table is searched to generate a triangular mesh to construct the surface of the object. For another example, based on the potential function of the field, the point cloud is converted into a dense surface model to generate a smooth surface to solve the problems of noise and incompleteness of the point cloud data. For another example, a sphere with a preset radius is used to roll on the surface of the point cloud to connect adjacent points to generate a mesh.

[0111] In one example, when missing areas appear in the point cloud of the spatial environment, interpolation or Laplace method is used to fill them and generate a complete surface; then, Laplace smoothing or bilateral smoothing algorithm is used to eliminate noise and fine textures on the model surface to obtain a smooth surface. After that, the normal vector of each spatial point is obtained. For example, the local plane of the current point is calculated using the point cloud of the field to obtain the normal vector.

[0112] In one example, the initial space model can be unfolded into a UV coordinate system, and then the aligned RGB image is used to add texture to the initial space model, and the occlusion area and perspective change are processed, and the optimal texture fragment is selected for mapping to improve visual realism.

[0113] In one example, the QEM (Quadric Error Metrics) algorithm can be used to adjust the polygon data of the initial spatial model to maintain the overall shape and details. Alternatively, the detail level of the model can be dynamically adjusted according to the distance to balance performance and effect.

[0114] In one example, a normal map may be generated for the simplified initial space model to ensure that the details of the high-resolution model are preserved.

[0115] In one embodiment, by acquiring a three-dimensional space model, the spatial structure of the space environment can be accurately reflected.

[0116] In step 12, the gaze point of the user in the three-dimensional space model is determined according to the eye direction vector and the three-dimensional space model.

[0117] In this step, the user's gaze point in the three-dimensional space model can be determined according to the eye direction vector and the three-dimensional space model. Figure 2 , including steps 21 to 24.

[0118] In step 21 , the spatial position and rotation angle of the head are obtained.

[0119] In this step, the wearable device may include a spatial attitude sensor, such as at least one of a gyroscope, a gravity sensor, or an accelerometer. The wearable device may use the spatial attitude sensor to collect the spatial position and spatial attitude of the wearable device. It is understandable that, considering that the wearable device is worn on the user's head, the spatial position and spatial attitude of the wearable device respectively represent the spatial position and spatial attitude of the user's head, that is, the spatial position and rotation angle of the head can be obtained.

[0120] Considering that the wearable device is pre-calibrated in the spatial scene before being worn, the wearable device stores a preset rotation matrix R, which can be obtained by Euler angle or quaternion conversion, and can represent the conversion relationship between the coordinate system of the wearable device and the world coordinate system. Therefore, in one embodiment, it can be understood that the above-mentioned head spatial position and rotation angle have been converted to the world coordinate system.

[0121] In step 22, the sight direction vector is converted to the three-dimensional space model according to a preset rotation matrix.

[0122] Considering that the wearable device is pre-calibrated in the spatial scene before being worn, a preset rotation matrix R is stored in the wearable device. The rotation matrix R can be obtained by Euler angle or quaternion conversion, which can represent the conversion relationship between the coordinate system of the wearable device and the world coordinate system.

[0123] After obtaining the sight direction vector, since the sight direction vector is a vector expression in the camera coordinate system, it can be recorded as G camera , combined with the above rotation matrix R, it is transformed into the world coordinate system, as shown in formula (3).

[0124] G world =R*G camera (3)

[0125] In formula (3), G world It represents the sight direction vector in the world coordinate system or the sight direction vector converted to the three-dimensional space model. R represents the rotation posture of the head relative to the world coordinate system, which is obtained by Euler angle or quaternion conversion and can be obtained by pre-calibrating the wearable device, that is, the preset rotation matrix.

[0126] In step 23, the viewpoint position is acquired according to the head space position, the rotation matrix and the preset offset.

[0127] In one embodiment, the viewpoint position can be obtained according to the head space position, the rotation matrix and the preset offset. Considering the deviation between the eye position and the head center position, the eye position needs to be corrected, as shown in formula (4).

[0128] Peye=T+R*T0 (4)

[0129] In formula (4), T0 represents the pre-measured offset between the eye position and the center of the head.

[0130] In this step, the eye sight line can be generated, that is, a ray can be generated from the eye, that is, the eye position Peye is used as the viewpoint position, that is, the starting point of the ray, and the sight direction vector G world As the ray direction, the ray expression is as shown in formula (5).

[0131] P(t)=Peye+t*G world (5)

[0132] In formula (5), t represents time, and its value is greater than or equal to 0.

[0133] In step 24, at least one intersection point between the eye ray and the three-dimensional space model is determined, and the nearest intersection point is used as the gaze point; or, when it is determined that the eye ray does not intersect with the three-dimensional space model, an invalid gaze point is obtained.

[0134] Considering that the three-dimensional space model is a three-dimensional grid model containing objects in the environment, in this step, the ray and the triangular face in the three-dimensional space model can be tested for intersection, which is also called collision detection.

[0135] In one example, the intersection point between the ray and each triangular face can be calculated to obtain a calculation result. When a triangular face intersects in the calculation result, an intersection point is obtained, and the intersection point is determined to be the gaze point; in another example, when two or more triangular faces intersect in the calculation result, that is, when there are two or more intersection points, the closest intersection point among the two or more intersection points is determined to be the gaze point Pgaze; in another example, when the ray does not intersect with the triangular face in the calculation result, that is, when there is no intersection point, it is determined that there is no intersection point or the intersection point is set to an invalid intersection point, and the value of the invalid intersection point can be set to the maximum distance or infinity or an illegal value.

[0136] In another example, the intersection of the ray and some triangular faces can be calculated to obtain the calculation result. The above-mentioned part of the triangular faces can be obtained first. For example, the ray has a direction, and the target plane passing through the starting point of the ray and perpendicular to the ray can be obtained; then the triangular faces on the first side and the second side of the target plane are determined, wherein the first side refers to the side to which the ray direction points, and the second side refers to the side opposite to the ray direction; then, the triangular faces on the second side are discarded, and the intersection of the ray and the triangular faces on the first side is calculated, and the intersection is determined as the gaze point Pgaze; when there are two or more triangular faces intersecting in the calculation result, that is, there are two or more intersection points, the closest intersection point among the two or more intersection points is determined as the gaze point Pgaze; when the ray does not intersect with the triangular face in the calculation result, that is, there is no intersection point, it is determined that there is no intersection point or the intersection point is set to an invalid intersection point, and the value of the invalid intersection point can be set to the maximum distance or infinity or an illegal value. In this way, the amount of calculation in the process of obtaining the gaze point can be reduced in this example.

[0137] In another example, considering that the eye gaze has a certain range, the target angle of the user's eye can be obtained in advance, for example, the value range of the target angle can be [5, 10] degrees, and the target angle represents the maximum angle of the user's eye visibility. After obtaining the ray, a rotation line is set based on the starting point of the ray, and the angle between the rotation line and the ray is the above target angle. Rotating the above rotation line can obtain a cone; then, the triangular face located inside the above cone is obtained, such as obtaining the projection of the cone in the space environment, and the triangular face falling within the projection is the target triangular face. After that, the intersection point of the ray and the target triangular face is obtained to obtain the calculation result. When the calculation result has only one intersection point, the intersection point is determined to be the gaze point Pgaze; when there are two or more triangular faces intersecting in the calculation result, the closest intersection point among the two or more intersection points is determined to be the gaze point Pgaze; when there is no intersection point in the calculation result, it is determined that there is no intersection point or the intersection point is set to an invalid intersection point, and the value of the invalid intersection point can be set to the maximum distance or infinity or an illegal value. In this way, the amount of calculation in the process of obtaining the gaze point can be reduced in this example.

[0138] In one example, an acceleration structure may be used to calculate the intersection of the ray and each triangular face to obtain a calculation result. The acceleration structure may use an acceleration structure such as a bounding volume hierarchy (BVH), a KD tree, or an octree to improve the efficiency of collision detection.

[0139] In this step, after obtaining the gaze point, the time t, the viewpoint position Peye, and the sight direction vector G worldand Pgaze as a data file in a preset data format. The above preset data format can be selected according to the specific scenario. For example, when the data file is a CSV file or a JSON file, the preset data format can be the format required by CSV or JSON.

[0140] In step 13, a movement trajectory of the user is generated according to all gaze points; the movement trajectory includes a motion trajectory and a gaze point trajectory.

[0141] In this step, the user's movement trajectory can be generated based on all gaze points, see Figure 3 , including steps 31 to 34.

[0142] In step 31, a region of interest in the three-dimensional space model is obtained.

[0143] In this step, the region of interest in the three-dimensional space model can be obtained. For example, key objects or regions in the three-dimensional space model can be identified and selected as the region of interest. For example, exhibits, artworks or display areas in a museum can be set in a spatial environment, which is not limited here.

[0144] In this step, the regions of interest may be grouped, named, and attributes may be set, and then each region of interest may be marked, and each region of interest may be assigned a unique identifier and metadata (such as name, category, description, etc.).

[0145] It is understandable that when generating a three-dimensional space model, the region of interest can be synchronously stored in the form of embedded data or external configuration files (such as JSON, XML files) to facilitate subsequent analysis and use.

[0146] In step 32, the region of interest where the gaze point is located is determined, and a gaze event is obtained.

[0147] In this step, when the gaze point is located in a certain region of interest, it can be recorded as a gaze event, which includes information such as time, user identification code, region of interest identification code, etc.

[0148] In step 33, statistical data of gaze events of each region of interest are obtained.

[0149] In this step, statistical data of gaze events of each region of interest may be obtained, for example, the statistical data may include: gaze duration, gaze count, average gaze duration, etc. of the region of interest, and the classification of statistical data may be selected according to the specific scene. It is understandable that after obtaining the statistical data, the statistical data may be saved in the above-mentioned recorded events.

[0150] Take the fixation duration as an example to obtain the fixation event statistics for each region of interest, see Figure 4 , which may include steps 41 to 44.

[0151] In step 41, a distance dij from the gaze point of each gaze event fi to the center point of the region of interest mj may be calculated, which is hereinafter referred to as event distance dij.

[0152] In step 42, an event threshold of the gaze event may be determined according to the time frame in which the gaze event occurs.

[0153] For example, first obtain the distance i from the viewpoint to the gaze point, and obtain the eye angle angle; then obtain the tangent value tan(angle rad / 2), where The above tangent value represents the calculated distance between the viewpoint and the gaze point.

[0154] Afterwards, the product of the distance distance i and the tangent value can be obtained, and twice the product is used as the event threshold, that is, the expression of the event threshold th i is shown in formula (6).

[0155]

[0156] In step 43, the event distance may be compared with the focus threshold. When the event distance is less than or equal to the event threshold, it is determined that the region of interest is in focus within the time frame. The time frame refers to the time frame collected by the wearable device.

[0157] In step 44, the duration of the focusing state of the region of interest may be obtained as the first statistical data of the gaze event of the region of interest.

[0158] In this step, the duration of the region of interest being in the focused state can be obtained, as shown in formula (7).

[0159]

[0160] In formula (7), ΔT i represents the duration of the focus of the triangle face mj in the i-th fixation event, which can be calculated based on the frame difference between the focus events and the interval of each time frame; I(d ij <th i ) represents an indicator function. If the triangle face m j In the i-th event, it is in the focused state (i.e., d ij <th i ), its value is 1, otherwise it is 0; n represents the number of times the triangle face mj is focused.

[0161] In this step, the duration that the above-mentioned region of interest is in the focused state may be used as the first statistical data of the gaze event of the region of interest.

[0162] In one example, the cumulative duration of all the regions of interest in the preset area can also be obtained. Then the total area of ​​all the regions of interest in the preset area is obtained. After that, the ratio of the cumulative duration and the total area is obtained as the second statistical data of the gaze event of the region of interest. In this way, in this example, the degree of attraction of the display space to the user's attention is measured by setting the gaze duration per unit area, thereby reflecting the difference in the degree of attraction of the user's attention between different display spaces, so as to help understand which areas are more able to attract the audience's attention and which areas are relatively weak, so as to facilitate the adjustment of exhibits in different display areas.

[0163] In step 34, a focus point trajectory matching the user's motion trajectory is generated according to the statistical data to obtain the movement trajectory.

[0164] In this step, the attention point trajectory matching the user's motion trajectory can be generated according to the statistical data to obtain the movement trajectory. The movement trajectory can be based on the user's gaze time or gaze times of the interested area at each spatial position, and the average gaze time of each interested area.

[0165] In this step, the above movement trajectory can be visualized, so that users can easily replay the tour route and gaze events.

[0166] In this step, recommendation information can be generated based on the above movement trajectory. For example, the area of ​​interest that the user has focused on the most times or the area of ​​interest with the longest gaze time can be obtained in the movement trajectory, and the exhibit information in the area of ​​interest can be redisplayed, or other objects related to the exhibits in the area of ​​interest, such as cultural and creative works of the exhibits, can be recommended to improve the user experience.

[0167] The following describes the above-mentioned moving trajectory acquisition method in combination with specific scenarios. Figure 5 ,include:

[0168] In one example, after the wearable device is worn, the wearable device can obtain the user's eyeball position and calibrate the user's head; then, the eyeball is calibrated again according to the calibrated head position. In addition, the viewpoint and sight direction vector of the eye can be obtained. The solution for obtaining the viewpoint and sight direction vector can refer to the content of the above example.

[0169] After the wearable device is worn, the wearable device can obtain a three-dimensional space model of the space scene while the user is moving. The solution for obtaining the three-dimensional space model can refer to the content of the above example. It is understandable that the three-dimensional space model can be updated with the incremental depth data and point cloud obtained during the user's movement, and the three-dimensional space model can be improved by integrating the depth data obtained at different times and perspectives to ensure the accuracy of the three-dimensional space model.

[0170] After generating the three-dimensional space model, the wearable device can obtain the virtual space anchor points, that is, the reference points set in the virtual space. The coordinate data of these reference points in the world coordinate system, or the relative coordinate data based on the reference origin (the world coordinate data of the reference origin is known) is known. In this way, the wearable device can calculate the coordinate data of all spatial points in the three-dimensional space model based on the above-mentioned virtual space anchor points.

[0171] Then, the ray can be obtained according to the viewpoint and the sight direction vector. The ray and the triangular face of the three-dimensional space model are used for collision detection to obtain the intersection point, i.e., the gaze point. In addition, at least one of the gaze point, head position, head rotation angle, head orientation, time, etc. is recorded and saved in the data file.

[0172] It should be noted that Figure 5 The left scheme can be understood as the data collection stage, where data files and three-dimensional space models are obtained; Figure 5 The solution on the right can be understood as the data processing stage to obtain a visualized movement trajectory.

[0173] In this example, the region of interest of the three-dimensional space model can be annotated. Then, the gaze events are generated using the gaze points and regions of interest in the data file; and the statistics of the gaze events are obtained. Then, the movement path is created based on the statistics, such as Figure 6 The movement trajectory 61 of a user is shown in the example. Figure 6 , the user forms a movement track 61 after moving in the three-dimensional space model 60. The movement track 61 reflects the user's position in each area (at different times) in the three-dimensional space model and the sight direction 62. Among them, the sight direction 62 represents the user's gaze area 63. In an example, the duration of the above-mentioned sight direction 62 also represents the duration of the user's gaze area 63.

[0174] In this example, the above movement path data can be visualized. Figure 7 , the user's movement trajectory in the three-dimensional space model. It should be noted that Figure 7Different color depths at different positions of the moving track in the figure indicate different statistical data of the user at the position. For example, the color depth at position 71 is darker than that at position 72. Taking position 71 as an example, the statistical data of the user at the position 71 for each area of ​​interest of the three-dimensional space model is as follows: Figure 8 See Figure 8 , user 81 is currently gazing at area 82, and previously (or later) gazing at area 84, and the color depth of area 82 is darker than that of area 84, indicating that user 81 has been gazing at area 82 for a longer time. The color depths of both area 82 and area 84 are darker than that of area 83, indicating that area 83 is not or less gazed at by user 81.

[0175] In this way, the visualization of the movement trajectory in the three-dimensional space model in this embodiment, as well as the statistical data reflecting the user's line of sight and the area of ​​interest, can simulate the specific behavior of the user during the movement. In other words, the visualization process in this example is the time t in the root data file. In the three-dimensional space model, the user's head position, line of sight direction, gaze point and other data are used to adjust the direction of the ray and the appearance time in the animation to simulate and reproduce the user's behavior. The ray mode is determined by the user's viewpoint direction, and the appearance time and change speed of the ray in the area of ​​interest are determined according to time. In this way, the movement trajectory can be reproduced for the user, and researchers can perform data analysis and decision-making.

[0176] Based on the above-mentioned method for obtaining a movement trajectory, the present disclosure also provides a device for obtaining a movement trajectory. Fig. 9 ,include:

[0177] A space model acquisition module 91 is used to acquire a three-dimensional space model of the space environment where the wearable device is located;

[0178] A direction vector acquisition module 92 is used to acquire a sight line direction vector of the eye of the user wearing the wearable device;

[0179] A gaze point acquisition module 93, configured to determine a gaze point of a user in the three-dimensional space model according to the eye direction vector and the three-dimensional space model;

[0180] The movement trajectory acquisition module 94 is used to generate the movement trajectory of the user according to all the gaze points; the movement trajectory includes a motion trajectory and a gaze point trajectory.

[0181] Optionally, the space model acquisition module includes:

[0182] A data acquisition submodule, used to acquire depth data, RGB image data and point cloud data collected by the wearable device;

[0183] An initial model acquisition submodule, used to align the spatial points in the depth data with the pixel points in the RGB image data to obtain an initial spatial model;

[0184] A normal vector acquisition submodule, used for acquiring the normal vectors of the polygons of each surface and each space point in the initial space model according to the point cloud data;

[0185] The space model acquisition submodule is used to perform normal mapping on the initial space model based on the normal vector to obtain a three-dimensional space model.

[0186] Optionally, the direction vector acquisition module includes:

[0187] A position acquisition submodule, used to acquire the pupil position and the reflection point position according to the eye image of the user's eye;

[0188] The vector acquisition submodule is used to determine the sight line direction vector based on a preset eyeball model, the pupil position and the reflection point position.

[0189] Optionally, the gaze point acquisition module includes:

[0190] The position angle acquisition submodule is used to obtain the head spatial position and rotation angle;

[0191] A direction conversion submodule, used for converting the sight direction vector into the three-dimensional space model according to a preset rotation matrix;

[0192] A viewpoint position acquisition submodule, used to acquire the viewpoint position according to the head space position, the rotation matrix and a preset offset;

[0193] The gaze point acquisition submodule is used to determine at least one intersection point between the eye ray and the three-dimensional space model, and use the nearest intersection point as the gaze point; or, when it is determined that the eye ray does not intersect with the three-dimensional space model, obtain an invalid gaze point.

[0194] Optionally, the movement trajectory acquisition module includes:

[0195] An area of ​​interest acquisition submodule is used to acquire an area of ​​interest in the three-dimensional space model;

[0196] A gaze event determination submodule, used to determine the region of interest where the gaze point is located, and obtain a gaze event;

[0197] A statistical data acquisition submodule is used to obtain statistical data of gaze events in each region of interest;

[0198] The movement trajectory acquisition submodule is used to generate a focus point trajectory matching the user movement trajectory according to the statistical data to obtain the movement trajectory.

[0199] Optionally, the statistical data acquisition submodule includes:

[0200] An event distance acquisition unit, used for acquiring the event distance from the gaze point to the center point of the region of interest in each gaze event;

[0201] An event threshold acquisition unit, used to determine an event threshold of the gaze event according to a time frame where the gaze event is located;

[0202] A focusing state determining unit, configured to determine that the region of interest is in a focusing state within the time frame when the event distance is less than or equal to the event threshold;

[0203] The first statistical data acquisition unit is used to acquire a duration during which the region of interest is in a focused state as first statistical data of a gaze event of the region of interest.

[0204] Optionally, the event threshold acquisition unit includes:

[0205] A distance acquisition subunit, used to acquire the distance from the viewpoint to the gaze point;

[0206] An eye perspective acquisition subunit, used to acquire the eye perspective;

[0207] A tangent value acquisition subunit, used to acquire a tangent value of half the viewing angle; the tangent value represents a calculated distance between the viewpoint and the gaze point;

[0208] The event threshold acquisition subunit is used to acquire the product of the distance and the tangent value, and take twice the product as the event threshold.

[0209] Optionally, the statistical data acquisition submodule further includes:

[0210] An accumulated duration acquisition unit, used to acquire the accumulated duration of all the regions of interest in a preset area;

[0211] A total area acquisition unit, used to acquire the total area of ​​all regions of interest in a preset area;

[0212] The second statistical data acquisition unit is used to acquire the ratio of the accumulated duration to the total area of ​​the region as the second statistical data of the gaze event of the region of interest.

[0213] It should be noted that the device shown in this embodiment matches the contents of the method embodiment, and reference may be made to the contents of the above method embodiment, which will not be repeated here.

[0214] The present disclosure also provides an electronic device, which may include a processor, a memory, and a display;

[0215] The memory is used to store a computer program executable by the processor;

[0216] Wherein, the processor is configured to execute the computer program in the memory to implement the above-mentioned method for obtaining a movement trajectory;

[0217] The display is used to visualize the three-dimensional space model and movement trajectory.

[0218] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0219] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A trajectory acquisition method, characterized in that: Applicable to a wearable device worn by a user, the method comprises: Obtaining a three-dimensional space model of the space environment in which the wearable device is located, and obtaining a sight line direction vector of the eyes of the user wearing the wearable device; Determining a gaze point of a user in the three-dimensional space model according to the eye direction vector and the three-dimensional space model; A movement trajectory of the user is generated according to all gaze points; the movement trajectory includes a motion trajectory and a gaze point trajectory.

2. The method according to claim 1, characterized in that Obtain the three-dimensional space model of the space environment where the wearable device is located, including: Acquire depth data, RGB image data, and point cloud data collected by the wearable device; Aligning the spatial points in the depth data with the pixel points in the RGB image data to obtain an initial spatial model; Acquire the polygons of each surface and the normal vectors of each space point in the initial space model according to the point cloud data; Perform normal mapping on the initial space model based on the normal vector to obtain a three-dimensional space model.

3. The method according to claim 1, characterized in that Obtaining the sight direction vector of the eye of the user wearing the wearable device, including: Obtaining a pupil position and a reflection point position according to an eye image of the user's eye; The sight line direction vector is determined based on a preset eyeball model, the pupil position and the reflection point position.

4. The method according to claim 1, characterized in that: Determining a gaze point of a user in the three-dimensional space model according to the eye direction vector and the three-dimensional space model includes: Get the head space position and rotation angle; Converting the sight direction vector to the three-dimensional space model according to a preset rotation matrix; Acquire the viewpoint position according to the head space position, the rotation matrix and the preset offset; Determine at least one intersection point between the eye ray and the three-dimensional space model, and use the nearest intersection point as the gaze point; or, when it is determined that the eye ray does not intersect with the three-dimensional space model, obtain an invalid gaze point.

5. The method according to claim 1, characterized in that Generating a movement trajectory of the user according to all gaze points includes: Acquire a region of interest in the three-dimensional space model; Determine the region of interest where the gaze point is located, and obtain a gaze event; Get statistics of fixation events for each region of interest; A focus point trajectory matching the user's motion trajectory is generated according to the statistical data to obtain the movement trajectory.

6. The method according to claim 5, characterized in that Get statistics on gaze events for each region of interest, including: Get the event distance from the fixation point in each fixation event to the center point of the region of interest; Determine an event threshold of the gaze event according to a time frame where the gaze event is located; When the event distance is less than or equal to the event threshold, determining that the region of interest is in a focused state within the time frame; The duration of the focusing state of the region of interest is obtained as the first statistical data of the gaze event of the region of interest.

7. The method according to claim 6, characterized in that Determining an event threshold of the gaze event according to a time frame where the gaze event is located includes: Get the distance from the viewpoint to the gaze point, and get the eye viewing angle; Obtaining a tangent value of half the viewing angle; the tangent value represents a calculated distance between the viewpoint and the gaze point; The product of the distance and the tangent value is obtained, and twice the product is used as the event threshold.

8. The method according to claim 6, characterized in that The method further comprises: Get the cumulative duration of all the regions of interest in the preset area; Get the total area of ​​all regions of interest in the preset area; A ratio of the accumulated duration to the total area of ​​the region is obtained as second statistical data of gaze events in the region of interest.

9. A trajectory acquisition device, characterized in that: A wearable device suitable for being worn by a user, the device comprising: A spatial model acquisition module is used to acquire a three-dimensional spatial model of the spatial environment in which the wearable device is located; A direction vector acquisition module, used to acquire a sight line direction vector of the eye of the user wearing the wearable device; A gaze point acquisition module, used for determining a gaze point of a user in the three-dimensional space model according to the eye direction vector and the three-dimensional space model; The movement trajectory acquisition module is used to generate the movement trajectory of the user according to all the gaze points; the movement trajectory includes a motion trajectory and a gaze point trajectory.

10. A trajectory acquisition system, characterized in that: include: Processor and memory; The memory is used to store a computer program executable by the processor; The processor is used to execute the computer program in the memory to implement the method according to any one of claims 1 to 7.

Images