User behavior feedback feature extraction method and system in virtual reality test scene

Abstract

The application discloses a user behavior feedback feature extraction method and system in a virtual reality test scene, comprising: firstly, obtaining a head rotation direction sequence and a line of sight landing point space coordinate sequence of a head-mounted display device, and obtaining a key trigger intensity sequence and a controller displacement trajectory sequence of a handheld control device; performing synchronous correlation processing through a cross-modal behavior response fusion network to generate a multi-source behavior response synchronous feature set; then, mining a gaze retention mode and a control rhythm mode, marking a focus of attention on original behavior data flow, and outputting a test scene behavior feedback feature set with a behavior feedback mode label, which is used for triggering a scene optimization response.

Description

Technical Field

[0001] This invention relates to the field of virtual reality interaction technology, and more specifically, to a method and system for extracting user behavior feedback features in a virtual reality testing scenario. Background Technology

[0002] Virtual reality testing scenarios typically collect user behavior data in an immersive environment using head-mounted displays and handheld control devices to evaluate scene interaction effects, user attention levels, and operational experience. Existing testing methods often focus on recording data in a single dimension, such as only counting gaze duration, head rotation angle, or number of button presses, which fails to comprehensively reflect the user's real-world behavioral feedback process in the virtual scene.

[0003] Because virtual reality interaction is characterized by multimodality, strong temporality, and high-frequency response, users' head posture, gaze point, controller displacement, and button operations often exhibit synchronous correlations. Traditional analysis methods lack the ability to align data streams from different devices in time, perform cross-modal fusion, and mine behavioral patterns, leading to problems such as fragmented features, delayed feedback, and insufficient basis for scene optimization in test results. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for extracting user behavior feedback features in virtual reality testing scenarios.

[0005] In a first aspect, embodiments of the present invention provide a method for extracting user behavior feedback features in a virtual reality testing scenario, the method comprising:

[0006] Acquire raw behavioral data stream transmitted by a head-mounted display device, wherein the raw behavioral data stream transmitted by the head-mounted display device includes a timestamp-aligned sequence of head rotation orientation and a sequence of spatial coordinates of gaze points;

[0007] Acquire the raw control data stream transmitted by the handheld control device, wherein the raw control data stream transmitted by the handheld control device includes a key trigger intensity sequence and a controller displacement trajectory sequence;

[0008] A preset cross-modal behavior response fusion network is invoked to perform multi-source behavior response synchronization association processing on the head rotation orientation sequence, the gaze point spatial coordinate sequence, the button trigger intensity sequence, and the controller displacement trajectory sequence, generating a multi-source behavior response synchronization feature set.

[0009] Based on the multi-source behavioral response synchronization feature set, user behavior feedback pattern mining processing is performed to generate a behavioral feedback response pattern description set, which includes gaze persistence pattern description unit and manipulation rhythm pattern description unit.

[0010] The original behavioral data stream transmitted by the head-mounted display device is annotated with focus based on the behavioral feedback response pattern description set to generate a test scenario behavioral feedback feature set with behavioral feedback pattern labels. The test scenario behavioral feedback feature set with behavioral feedback pattern labels is then transmitted to the test analysis terminal to trigger a scenario optimization response.

[0011] Secondly, embodiments of the present invention provide a user behavior feedback feature extraction system in a virtual reality testing scenario, including at least one service node;

[0012] The service node includes a storage unit and a computing unit; the storage unit is used to store program code; the computing unit is used to run the program code to execute the user behavior feedback feature extraction method in the virtual reality test scenario described in the first aspect.

[0013] Compared with existing technologies, the beneficial effects provided by this invention include: Using the user behavior feedback feature extraction method and system disclosed in this invention for virtual reality testing scenarios, the invention acquires the original behavior data stream transmitted from a head-mounted display device, including a timestamp-aligned head rotation orientation sequence and a gaze point spatial coordinate sequence; acquires the original control data stream transmitted from a handheld control device, including a button trigger intensity sequence and a controller displacement trajectory sequence; invokes a cross-modal behavior response fusion network to synchronously correlate the four types of sequences—head rotation orientation, gaze point spatial coordinates, button trigger intensity, and controller displacement trajectory—to generate a multi-source behavior response synchronization feature set; performs behavior feedback pattern mining based on the synchronization feature set to generate a behavior feedback response pattern description set including gaze persistence pattern description and control rhythm pattern description; and annotates the focus of attention on the original behavior data stream of the head-mounted display device according to this description set, generating a test scenario behavior feedback feature set with feedback pattern labels and transmitting it to the test analysis terminal to trigger scene optimization response. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as limiting the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating the steps of a method for extracting user behavior feedback features in a virtual reality testing scenario provided in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the cross-modal behavior response fusion network provided in an embodiment of the present invention;

[0017] Figure 3 This is a schematic diagram of a behavior feedback response pattern mining structure based on a multi-source behavior response synchronization feature set provided in an embodiment of the present invention;

[0018] Figure 4 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0020] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] In order to solve the technical problems mentioned in the background art Figure 1 This is a flowchart illustrating the user behavior feedback feature extraction method in a virtual reality testing scenario provided in this embodiment of the disclosure. The following is a detailed description of the user behavior feedback feature extraction method in a virtual reality testing scenario.

[0022] Step S201: Obtain the raw behavioral data stream transmitted by the head-mounted display device. The raw behavioral data stream transmitted by the head-mounted display device includes a timestamp-aligned sequence of head rotation orientation and a sequence of spatial coordinates of the gaze point.

[0023] Step S202: Obtain the raw control data stream transmitted by the handheld control device, wherein the raw control data stream transmitted by the handheld control device includes a key trigger intensity sequence and a controller displacement trajectory sequence;

[0024] Step S203: Invoke a preset cross-modal behavior response fusion network to perform multi-source behavior response synchronization association processing on the head rotation orientation sequence, the gaze point spatial coordinate sequence, the button trigger intensity sequence and the controller displacement trajectory sequence, and generate a multi-source behavior response synchronization feature set.

[0025] Step S204: Based on the multi-source behavior response synchronization feature set, perform user behavior feedback pattern mining processing to generate a behavior feedback response pattern description set, which includes gaze persistence pattern description unit and control rhythm pattern description unit.

[0026] Step S205: Perform focus annotation processing on the raw behavior data stream transmitted by the head-mounted display device according to the behavior feedback response mode description set, generate a test scene behavior feedback feature set with behavior feedback mode labels, and transmit the test scene behavior feedback feature set with behavior feedback mode labels to the test analysis terminal to trigger scene optimization response.

[0027] In this embodiment of the invention, for example, the server acts as the behavioral analysis execution entity in the virtual reality testing platform. It receives multimodal behavioral data uploaded from a head-mounted display device and a handheld control device, and extracts the user's attention state, manipulation state, and feedback characteristics in the virtual reality testing scenario. In a product assembly-type virtual reality testing scenario, the user wears a head-mounted display device to observe a virtual assembly table, tool rack, prompt panel, and parts to be operated, while simultaneously performing interactive operations such as grasping, moving, rotating, and confirming using a handheld control device. The server first acquires the raw behavioral data stream transmitted by the head-mounted display device. This raw behavioral data stream includes a timestamp-aligned sequence of head rotation orientation and a sequence of spatial coordinates of the gaze point. Specifically, when the user turns from the assembly table to the prompt panel, the head-mounted display device continuously uploads data on changes in head yaw angle, pitch angle, and roll angle, and simultaneously uploads the three-dimensional coordinates of the gaze point in the virtual scene coordinate system. The server organizes the head rotation changes, gaze movement trajectory, and gaze dwell time based on the timestamps to determine whether the user is truly paying attention to a virtual object or prompt information.

[0028] The server further acquires the raw control data stream transmitted by the handheld control device. This raw control data stream includes a sequence of button trigger intensity and a sequence of controller displacement trajectories. Specifically, when the user presses the controller trigger button to grab the virtual wrench and moves it to the nut position, the handheld control device uploads the process of the button intensity changing from low to high and then gradually released, while simultaneously uploading the controller's movement trajectory in virtual space. The server manages this control data and the head-mounted display device data on the same timeline, enabling subsequent analysis to simultaneously reflect where the user "looks" and "how they operate."

[0029] In this embodiment of the invention, to ensure that the data uploaded by the head-mounted display device and the handheld control device can be fused and processed on a unified time scale, the server performs unified sampling, timestamp correction, and normalization processing on the original behavior data stream and the original control data stream before calling the cross-modal behavior response fusion network. Specifically, let the head rotation orientation sequence be... ,in , Indicates the yaw angle. Indicates pitch angle, Indicates the roll angle. Represents the sampling timestamp; let the spatial coordinate sequence of the line-of-sight landing point be... ,in Let the key trigger strength sequence be... ,in , Indicates the key trigger strength; let the controller displacement trajectory sequence be... ,in The server resamples H, G, B, and C at a preset fusion sampling frequency fs. The preset fusion sampling frequency fs can be set to 60Hz, 90Hz, or the same as the refresh rate of the head-mounted display device. For data points with incompletely consistent sampling timestamps, the server uses linear interpolation to generate the corresponding data value at the sampling time; for data points with consecutive data loss durations less than a preset data loss duration threshold... For data segments, the server uses adjacent valid sampling points for interpolation repair; for data segments with a continuous loss duration of not less than [amount missing], [the server performs interpolation repair]. For any given data segment, the server marks the corresponding time period as an invalid behavior segment and skips this invalid behavior segment in subsequent synchronization and association processing. The server further normalizes the resampled continuous features. For any continuous feature sequence... The server calculates the normalized eigenvalues ​​as follows: Where μX is the mean of the feature sequence X, σX is the standard deviation of the feature sequence X, and ε is a constant to prevent the denominator from being zero. The normalized head rotation orientation sequence, gaze point spatial coordinate sequence, button trigger intensity sequence, and controller displacement trajectory sequence are used as input data for the cross-modal behavior response fusion network.

[0030] After receiving the data, the server invokes a pre-defined cross-modal behavior response fusion network to perform synchronous correlation processing on the head rotation orientation sequence, gaze placement spatial coordinate sequence, key trigger intensity sequence, and controller displacement trajectory sequence of multi-source behavior responses. The server first segments the continuous data into head rotation primitive segments, gaze persistence primitive segments, saccade jump primitive segments, force application gradient primitive segments, relaxation recovery primitive segments, and control trajectory primitive segments. Then, based on the overlap of the time windows, it aligns the head rotation primitive segments with the gaze persistence primitive segments to form head-eye synchronization coupling primitive pairs, and aligns the force application gradient primitive segments with the control trajectory primitive segments to form hand-control synchronization coupling primitive pairs. For example, when a user stably gazes at a nut hole and simultaneously presses the grasp button, the server identifies the synchronization relationship between head orientation, gaze persistence, pressure application, and controller approach to the target. It then generates a fused behavior representation vector through cross-channel interaction and feature mapping, and encapsulates this vector to obtain a multi-source behavior response synchronization feature set.

[0031] The server mines user behavior feedback patterns based on a multi-source behavior response synchronization feature set, generating a behavior feedback response pattern description set. The server statistically analyzes the gaze persistence duration from head-eye synchronization coupling primitive pairs and distinguishes between gaze persistence-type coupling primitive pairs and gaze saccade-type coupling primitive pairs based on a preset gaze persistence duration threshold. For gaze persistence-type coupling primitive pairs, the server extracts the small angular displacement fluctuation features of the head during gaze and performs frequency domain transformation to obtain the dominant frequency component of head micro-movement and its energy amplitude. Simultaneously, the server extracts the spatial coordinate drift trajectory of the gaze point and calculates the drift envelope area as a description of gaze stability features. For example, when a user views a warning panel for a long time, the head remains stable but exhibits slight swaying, and the gaze drifts within a small range within the text area; the server generates a gaze persistence pattern description unit based on this. The server also extracts the duration of the rising and falling edges of the press from the hand-controlled synchronization coupling primitive pairs, constructs a press control rhythm pattern vector, analyzes its dominant rhythm cycle, and generates a control rhythm pattern description unit.

[0032] The server then annotates the raw behavioral data stream with focus of attention based on the behavioral feedback response pattern description set. The server generates a gaze focus indication vector based on the gaze persistence pattern description unit and aligns the dominant rhythmic period in the control rhythm pattern description unit with this indication vector in time to determine the overlapping period between gaze persistence events and control rhythmic events. When a user continuously gazes at a virtual component and synchronously performs stable control, the server marks the corresponding head rotation orientation sequence segment and gaze placement spatial coordinate sequence segment as high-attention behavioral feedback segments; when the user's gaze briefly sweeps across an area without forming an effective control, the server marks it as a low-attention behavioral feedback segment. For high-attention behavioral feedback segments, the server further maps the gaze placement coordinates back to virtual scene objects to obtain a set of high-attention virtual object identifiers, and generates a virtual object attention priority sequence based on the cumulative gaze duration of each object.

[0033] Finally, the server combines the virtual object attention priority sequence, scene functional area categories, and head micro-motion feature descriptions to construct a scene area attention behavior association mapping table. It then encapsulates high-attention behavior feedback segments, low-attention behavior feedback segments, virtual object attention priority sequences, and the scene area attention behavior association mapping table into a test scene behavior feedback feature set labeled with behavior feedback patterns. The server transmits this feature set to the test analysis terminal, which uses it to identify virtual objects that users focus on but find difficult to operate, scene areas with insufficient attention, and interactive elements with abnormal control rhythms. The terminal then triggers scene optimization responses, such as enhancing key prompts, adjusting interaction sensitivity, reducing the abruptness of interfering objects, or optimizing the layout of virtual objects, thereby achieving adaptive optimization of the virtual reality test scene based on real user behavior feedback.

[0034] Please refer to the following: Figure 2 , Figure 2 This is a schematic diagram of the cross-modal behavior response fusion network provided in an embodiment of the present invention. In this embodiment, the preset cross-modal behavior response fusion network is invoked to perform synchronous association processing on the head rotation orientation sequence, the gaze point spatial coordinate sequence, the button trigger intensity sequence, and the controller displacement trajectory sequence to generate a multi-source behavior response synchronization feature set. This can be implemented through the following example.

[0035] The head rotation orientation sequence is subjected to motion rhythm segmentation processing, and the head rotation orientation sequence is divided into a set of head rotation primitive segments according to the inflection point of the head rotation direction change. Each head rotation primitive segment in the set of head rotation primitive segments has a single directional attribute.

[0036] The spatial coordinate sequence of the gaze point is segmented by gaze stability processing, and the spatial coordinate sequence of the gaze point is divided into a set of gaze dwell primitive segments and a set of saccade jump primitive segments according to the dwell time of the spatial coordinate of the gaze point.

[0037] The button trigger intensity sequence is segmented by pressing state, and the button trigger intensity sequence is divided into a set of force-gradient primitive segments and a set of relaxation-recovery primitive segments according to the curvature of the button trigger intensity change.

[0038] The controller displacement trajectory sequence is subjected to spatial trajectory segmentation processing, and the controller displacement trajectory sequence is divided into a set of control trajectory primitive segments according to the abrupt curvature change points of the controller displacement trajectory.

[0039] The set of head rotation primitive fragments and the set of gaze persistence primitive fragments are input into the first synchronization alignment layer of the cross-modal behavior response fusion network. The temporal coupling degree of the head rotation primitive fragments and the gaze persistence primitive fragments is calculated based on the temporal window overlap, and a set of head-eye synchronization coupling primitive pairs is generated.

[0040] The force-gradient primitive fragment set and the control trajectory primitive fragment set are input into the second synchronization alignment layer of the cross-modal behavior response fusion network. The temporal coupling degree between the force-gradient primitive fragment and the control trajectory primitive fragment is calculated based on the time window overlap, and a set of manual control synchronization coupling primitive pairs is generated.

[0041] The set of head-eye synchronization coupling primitives and the set of hand-control synchronization coupling primitives are input into the cross-channel interaction layer of the cross-modal behavior response fusion network to calculate the co-occurrence frequency of head-eye coupling primitive segments and hand-control coupling primitive segments on the time axis and generate a cross-channel behavior response co-occurrence matrix.

[0042] The cross-channel behavior response co-occurrence matrix is ​​subjected to high-dimensional mapping processing. The cross-modal behavior response fusion network is used to project the cross-channel behavior response co-occurrence matrix onto a high-dimensional behavior feature space to generate a high-dimensional behavior response distribution vector.

[0043] The high-dimensional behavior response distribution vector is subjected to feature decoupling processing, which splits the high-dimensional behavior response distribution vector into head-eye collaborative sub-vectors and hand-control collaborative sub-vectors along the channel dimension, and performs cross-modal attention weighted fusion on the head-eye collaborative sub-vectors and hand-control collaborative sub-vectors to generate a fused behavior representation vector.

[0044] The fused behavior representation vector is associated and encapsulated with the temporal boundary information of the head rotation primitive fragment set, the gaze persistence primitive fragment set, the force gradient primitive fragment set, and the control trajectory primitive fragment set to generate the multi-source behavior response synchronization feature set.

[0045] In this embodiment of the invention, for example, after receiving data uploaded by the head-mounted display device and the handheld control device, the server invokes a preset cross-modal behavior response fusion network to perform synchronous association processing. The server first performs motion rhythm segmentation on the head rotation orientation sequence, continuously reading the changes in yaw, pitch, and roll angles of the user in the virtual assembly test scenario, and then segments the data according to the inflection points of the head rotation direction changes. For example, if the user first turns from the virtual assembly table to the left tool rack, and then from the tool rack to the upper warning panel, the server segments the processes of "turning left," "looking up," and "returning to center" into head rotation primitive segments, ensuring that each segment has a single directional attribute.

[0046] The server then performs gaze stability segmentation on the spatial coordinate sequence of the user's gaze point. When the user's gaze lingers on the nut hole, tool icon, or prompt text for a certain period of time, the server segments this coordinate sequence into gaze persistence primitive segments; when the user's gaze quickly jumps from the nut hole to the wrench icon, the server segments this rapid migration process into saccade jump primitive segments. Simultaneously, the server segments the button trigger intensity sequence into press state segments, identifying gradually increasing force application primitive segments based on the curvature of the button trigger intensity change, and gradually decreasing relaxation recovery primitive segments when the user releases the trigger button. For the controller displacement trajectory sequence, the server performs spatial trajectory segmentation based on the trajectory curvature abrupt change points, segmenting actions such as approaching parts in a straight line, rotating the nut in an arc, and avoiding obstacles in a zigzag pattern into control trajectory primitive segments.

[0047] In this embodiment of the invention, the server generates various primitive fragments according to the following rules. For the head rotation orientation sequence, the server calculates the head angular velocity between adjacent sampling points. ,in, This represents the vector norm composed of changes in yaw, pitch, and roll angles. When the direction of the head angular velocity changes, and the magnitude of the angular velocity change exceeds a preset angular velocity inflection point threshold θω, the server identifies the corresponding sampling point as the inflection point of the head rotation direction change, and defines the data segment between two adjacent head rotation direction change inflection points as a head rotation primitive segment. For the spatial coordinate sequence of the line-of-sight landing point, the server calculates the line-of-sight movement speed between adjacent sampling points. During a continuous time period When the gaze velocity is less than the preset gaze velocity threshold θg, and the duration of the continuous time period is not less than the preset gaze dwell time threshold Tdwell, the server determines the continuous time period as a gaze dwell primitive segment; when When the distance between the starting and ending spatial coordinates of the gaze point and the ending spatial coordinates of the gaze point within the given time period is not less than θg, and the distance between these coordinates is greater than the preset saccade distance threshold Dsaccade, the server identifies the given time period as a saccade jump primitive segment. For the key trigger intensity sequence, the server calculates the first-order rate of change of the key trigger intensity. and second-order rate of change .when When the value exceeds a preset rising edge threshold θrise within a continuous time period, the server defines that continuous time period as a force-gradient primitive segment; when r i If the value is less than the preset falling edge threshold θfall within a continuous time period, the server will terminate the continuous... The interval is determined to be a relaxation recovery primitive segment. For the controller displacement trajectory sequence, the server calculates the local trajectory curvature based on three consecutive controller spatial location points. i When the change in trajectory curvature corresponding to adjacent sampling points When the value exceeds the preset curvature change threshold θκ, the server determines the sampling point as the control trajectory segmentation point and the trajectory data segment between adjacent control trajectory segmentation points as the control trajectory primitive segment.

[0048] Subsequently, the server inputs the head rotation primitive fragment set and the gaze persistence primitive fragment set into the first synchronization alignment layer of the cross-modal behavior response fusion network to calculate their overlap within a time window. When the user's head is stably facing the assembly table and their gaze is synchronously fixed on the target nut, the server determines that the head rotation primitive fragment and the gaze persistence primitive fragment have a high temporal coupling degree and generates a set of head-eye synchronization coupling primitive pairs. The server then inputs the force gradient primitive fragment set and the control trajectory primitive fragment set into the second synchronization alignment layer to calculate the temporal coupling degree between the press enhancement process and the controller movement process. For example, if the user presses the grasp button while moving the controller to the wrench position, the server generates the corresponding set of hand control synchronization coupling primitive pairs.

[0049] In this embodiment of the invention, both the first synchronization alignment layer and the second synchronization alignment layer complete the synchronization alignment process based on the overlap of time windows and the degree of time coupling. Let the first primitive fragment be... The start and end times are Second primitive fragment The start and end times are The overlap of their time windows for: .when When the overlap is not less than the preset synchronization overlap threshold θoverlap, the server determines... and The candidate synchronization alignment conditions are met. Further, the server calculates the temporal coupling degree based on the time window overlap, the segment center time difference, and the segment duration difference. : ,in, and They are respectively and The central moment, and They are respectively and The duration of the term is defined by α, β, and γ, which are weighting coefficients, with α + β + γ = 1, and τ and τd are time scale parameters. For the same head rotation primitive segment, if multiple gaze-resident primitive segments satisfy the candidate synchronization alignment conditions, the server selects the temporal coupling degree. The largest gaze persistence primitive segment and the head rotation primitive segment form a head-eye synchronization coupling primitive pair. For the force gradient primitive segment and the control trajectory primitive segment, the server uses the same time window overlap and time coupling degree calculation method to generate the manual control synchronization coupling primitive pair.

[0050] The server further inputs the sets of head-eye synchronization coupling primitive pairs and hand-control synchronization coupling primitive pairs into the cross-channel interaction layer, statistically analyzing the co-occurrence frequency of head-eye coupling behaviors and hand-control coupling behaviors on a unified time axis to generate a cross-channel behavior response co-occurrence matrix. This matrix records the composite behavioral relationships of the user: "looking at the target part, stable gaze, synchronized force application, and movement controller." The server performs high-dimensional mapping on this co-occurrence matrix through a fully connected mapping layer, projecting it onto a high-dimensional behavior feature space to generate a high-dimensional behavior response distribution vector. Subsequently, the server decouples this vector, splitting it along the channel dimension into head-eye collaborative sub-vectors and hand-control collaborative sub-vectors. A cross-modal attention mechanism is then used to weightedly fuse the two types of sub-vectors, highlighting the correlation between the user's actual focus on the target and the effective control action, generating a fused behavior representation vector. Finally, the server associates and encapsulates the fused behavior representation vector with the start and end times, duration, and segment identifiers of head rotation primitive segments, gaze persistence primitive segments, force gradient primitive segments, and control trajectory primitive segments to form a multi-source behavior response synchronization feature set, providing a structured data foundation for subsequent gaze persistence analysis, control rhythm mining, and attention focus annotation.

[0051] In this embodiment of the invention, the cross-modal behavior response fusion network includes an input encoding layer, a first synchronization alignment layer, a second synchronization alignment layer, a cross-channel interaction layer, a fully connected mapping layer, a cross-modal attention fusion layer, and an output encapsulation layer. The input encoding layer encodes each primitive segment into a fixed-dimensional segment feature vector. For a head rotation primitive segment, the input encoding layer extracts the average angular velocity, maximum angular velocity, angular velocity variance, rotation direction unit vector, and duration to generate a head rotation segment vector vh. For a gaze persistence primitive segment, the input encoding layer extracts the gaze duration, mean coordinates of the gaze point, gaze point variance, drift path length, and average drift velocity to generate a gaze segment vector vg. For a force gradation primitive segment, the input encoding layer extracts the initial intensity, peak intensity, rising edge duration, average force application rate, and force application rate variance to generate a button segment vector vb. For a control trajectory primitive segment, the input encoding layer extracts the trajectory length, average velocity, maximum velocity, mean curvature, curvature variance, and trajectory direction vector to generate a trajectory segment vector vc. The first synchronization alignment layer receives vh and vg, and generates a head-eye synchronization coupling primitive pair vector qeg based on the temporal coupling degree: qeg=[vh,vg,Couple(h,g)]. The second synchronization alignment layer receives vb and vc, and generates a hand-controlled synchronization coupling primitive pair vector qhc based on the temporal coupling degree: qhc=[vb,vc,Couple(b,c)]. The cross-channel interaction layer is used to construct the cross-channel behavior response co-occurrence matrix M. Let the set of head-eye synchronization coupling primitive pairs be... The set of manually controlled synchronous coupling primitive pairs is Then the elements in the co-occurrence matrix M for: ,in, This represents the window overlap between the i-th head-eye synchronization coupling primitive pair and the j-th hand-controlled synchronization coupling primitive pair on the time axis. This indicates the temporal coupling degree between the two. The fully connected mapping layer maps the co-occurrence matrix M to a high-dimensional behavioral response distribution vector z: z = ReLU(Wm·vec(M) + bm), where vec(M) represents a one-dimensional vector obtained by vectorizing the co-occurrence matrix M, Wm represents the mapping weight matrix, bm represents the bias term, and ReLU represents the nonlinear activation function. The cross-modal attention fusion layer decomposes the high-dimensional behavioral response distribution vector z into a head-eye co-vector ze and a hand-control co-vector zh, and calculates the attention weights respectively: ae = softmax(Wae·ze + bae); ah = softmax(Wah·zh + bah). The fused behavioral representation vector f is generated as follows: ,in, [ ] represents element-wise multiplication, and [ ] represents vector concatenation. In one optional implementation, the cross-modal behavior response fusion network uses supervised training to determine network parameters. The server acquires historical virtual reality test data, which includes head rotation orientation sequence, gaze point spatial coordinate sequence, button trigger intensity sequence, controller displacement trajectory sequence, and manually or rule-labeled high-attention segments, low-attention segments, abnormal control segments, and visual fatigue segments. The server uses the fused behavior representation vector f as classification input, the labeling results as training labels, and the cross-entropy loss function to train the network parameters: Where y represents the true label and ŷ represents the network-predicted label. Let represent the set of network parameters, and λ represent the regularization coefficient. In another optional implementation, when manually labeled data is unavailable, the server uses an unsupervised parameter setting method, determining Wm, Wae, and Wah by minimizing the principal component directions of historical samples or by minimizing the reconstruction error of the autoencoder, so that the fused behavior representation vector f retains the main co-occurrence relationships between multi-source behavior responses. After training or parameter setting, the cross-modal behavior response fusion network receives multimodal behavior data from the current virtual reality test scenario during the inference phase and outputs the fused behavior representation vector and its corresponding temporal boundary information.

[0052] Please refer to the following: Figure 3 , Figure 3 This is a schematic diagram of a behavior feedback response pattern mining structure based on a multi-source behavior response synchronization feature set provided in an embodiment of the present invention. In this embodiment, the user behavior feedback pattern mining process based on the multi-source behavior response synchronization feature set to generate a behavior feedback response pattern description set can be implemented through the following example.

[0053] Extract the head-eye synchronization coupling primitive pair set from the multi-source behavior response synchronization feature set, perform gaze persistence duration statistical processing on each head-eye synchronization coupling primitive pair in the head-eye synchronization coupling primitive pair set, and calculate the time span of the gaze persistence primitive segment.

[0054] Based on the comparison between the time span of the gaze persistence primitive segment and the preset gaze persistence duration threshold, the head-eye synchronization coupling primitive pair set is divided into a gaze persistence type coupling primitive pair subset and a gaze saccade type coupling primitive pair subset.

[0055] For each gaze-pause type coupling primitive pair in the subset of gaze-pause type coupling primitive pairs, head micro-motion feature extraction processing is performed to extract the small angular offset fluctuation features of the head rotation orientation sequence during gaze-pause.

[0056] The minute angular offset fluctuation characteristics are transformed from the time domain to the frequency domain to obtain the head micro-motion frequency domain energy distribution.

[0057] Based on the energy distribution in the frequency domain of head micro-motion, the dominant frequency component of head micro-motion and the corresponding energy amplitude of the dominant frequency component of head micro-motion are extracted. The dominant frequency component of head micro-motion and the corresponding energy amplitude of the dominant frequency component of head micro-motion are used as the description of head micro-motion features in the gaze persistence state.

[0058] For each gaze-pause type coupling primitive pair in the subset of gaze-pause type coupling primitive pairs, the gaze-pause point drift feature extraction process is performed to extract the gaze-pause point spatial coordinate drift trajectory of the aligned gaze-pause spatial coordinate sequence.

[0059] The drift range of the spatial coordinate drift trajectory of the gaze point is quantified, the drift envelope area of ​​the spatial coordinate drift trajectory of the gaze point is calculated, and the drift envelope area is used as a description of the gaze stability feature under the gaze persistence state.

[0060] The head micro-movement feature description and the gaze stability feature description are spliced ​​together to generate the gaze persistence mode description unit.

[0061] Extract the set of manual synchronization coupling primitives from the set of multi-source behavioral response synchronization features, perform pressure rhythm analysis on each manual synchronization coupling primitive pair in the set of manual synchronization coupling primitives, and extract the duration of the rising edge of the pressure application gradient primitive segment and the duration of the falling edge of the pressure relaxation recovery primitive segment.

[0062] A press control rhythm pattern vector is constructed based on the alternating sequence of the duration of the press rising edge and the duration of the press falling edge. The press control rhythm pattern vector is then subjected to periodic analysis to extract the dominant rhythm period of the press control rhythm pattern vector. The dominant rhythm period is then used as the description unit of the control rhythm pattern.

[0063] In an embodiment of the invention, for example, after generating a multi-source behavioral response synchronization feature set, the server continues to mine user behavior feedback patterns from the synchronization coupling results within it. The server first extracts a set of head-eye synchronization coupling primitive pairs from the multi-source behavioral response synchronization feature set, and then reads the start and end times of the gaze persistence primitive segment in each head-eye synchronization coupling primitive pair. Taking a virtual assembly test scenario as an example, when the user observes the target nut, torque indicator panel, and error warning box on the assembly table, the server respectively counts the duration of the gaze's focus within the aforementioned object areas and calculates the time span of each gaze persistence primitive segment.

[0064] The server compares the time span of each gaze persistence primitive segment with a preset gaze persistence duration threshold. When the user's gaze lingers on the target nut hole for more than the threshold, the server assigns the corresponding head-eye synchronization coupling primitive pair to a subset of gaze persistence coupling primitive pairs; when the user's gaze only quickly sweeps over the tool rack icon and immediately turns to the assembly table, the server assigns the corresponding head-eye synchronization coupling primitive pair to a subset of gaze saccade coupling primitive pairs. Through this assignment, the server distinguishes between the user's effective attention behavior and brief browsing behavior.

[0065] For the gaze-holding coupled primitive pair subset, the server further extracts the minute angular offset fluctuation features in the head rotation orientation sequence during gaze-holding. Specifically, when a user views the red warning text on the warning panel for an extended period, the overall head orientation remains stable, but the yaw and pitch angles exhibit slight back-and-forth variations. The server treats these minor variations as head micro-motion data and performs a frequency domain transformation, converting the angular fluctuations in the time domain into a frequency domain energy distribution. The server then extracts the highest-energy dominant frequency component of the head micro-motion and its corresponding energy amplitude from the head micro-motion frequency domain energy distribution, using both as a description of the head micro-motion features in the gaze-holding state to characterize the user's level of recognition, hesitation, or tension during stable gaze.

[0066] The server also extracts gaze-point drift features for each gaze-lingering coupled primitive pair. When a user gazes at the virtual wrench icon, the gaze point is not fixed at a single point, but rather forms a small drift trajectory between the icon boundary, text description, and highlighted outline. The server extracts the time-aligned spatial coordinate sequence of the gaze point, generates the gaze-point spatial coordinate drift trajectory, quantifies the drift range of this trajectory, and calculates its drift envelope area. A smaller drift envelope area indicates stable user gaze and clear target recognition; a larger drift envelope area indicates wandering user gaze and unstable target of attention. The server uses the drift envelope area as a feature description of gaze stability and concatenates it with the aforementioned head micro-motion main frequency component and energy amplitude to generate a gaze-lingering mode description unit.

[0067] Simultaneously, the server extracts a set of manual control synchronization coupling primitive pairs from the multi-source behavioral response synchronization feature set, and performs pressure rhythm analysis on each manual control synchronization coupling primitive pair. When the user continuously grabs, rotates, and releases virtual parts, the server extracts the duration of the rising edge of the pressure gradient primitive segment and the duration of the falling edge of the pressure relaxation recovery primitive segment in the key trigger intensity curve. The server arranges multiple rising edge durations and falling edge durations into an alternating sequence according to time order to construct a pressure control rhythm pattern vector. Subsequently, the server performs periodic analysis on this vector to extract the dominant rhythmic cycle of the user in continuous interaction. For example, the pressure cycle of skilled users is stable and highly repeatable, while the pressure cycle of disturbed users is prolonged or interrupted. The server uses this dominant rhythmic cycle as the control rhythm pattern description unit, and together with the gaze persistence pattern description unit, forms a behavioral feedback response pattern description set.

[0068] In this embodiment of the invention, when the server performs frequency domain transformation on the minute head angular displacement fluctuation characteristics during gaze persistence, a Fast Fourier Transform can be used. Let the sequence of head micro-movements during the gaze persistence period be δH={δh1,δh2,…,δhN}, where… Hdwell represents the sequence of head rotation orientations during the fixation period, and its frequency domain energy distribution is... The server identifies the frequency point with the largest energy amplitude in E(k) as the dominant head-motion frequency component fmain, and defines E(fmain) as the energy amplitude corresponding to the dominant head-motion frequency component. For the gaze-point spatial coordinate drift trajectory, the server projects the gaze-point spatial coordinates during the gaze persistence period onto the current user's view plane, obtaining a two-dimensional gaze drift point set. The server performs convex hull calculation on P to obtain the envelope polygon Hull(P), and calculates the drift envelope area according to the following polygon area formula. When A is less than the preset stable area threshold θA, the server determines that the gaze persistence segment has high visual stability; when A is not less than θA, the server determines that the gaze persistence segment has enhanced visual drift. For the press control rhythm pattern vector, let the sequence consisting of alternating press rising edge duration and falling edge duration be... The server calculates its autocorrelation function Corr(k): The server searches for the maximum peak value k* of Corr(k) within the preset period search range [kmin, kmax], and determines Tmain=k* / fs as the dominant rhythm period. When the maximum peak value is less than the preset period confidence threshold θcorr, the server marks the press-operated rhythm mode as an unstable rhythm mode.

[0069] In this embodiment of the invention, the step of performing focus annotation processing on the raw behavior data stream transmitted by the head-mounted display device according to the behavior feedback response pattern description set to generate a test scenario behavior feedback feature set with behavior feedback pattern labels can be implemented through the following example.

[0070] The head micro-movement feature description and the gaze stability feature description contained in the gaze persistence mode description unit of the behavior feedback response mode description set are fused to generate a gaze focus indication vector.

[0071] The dominant rhythmic period contained in the control rhythmic pattern description unit of the behavioral feedback response pattern description set is time-aligned with the gaze attention focus indicator vector to determine the overlapping time period of gaze persistence event and control rhythmic event on the time axis.

[0072] The head rotation orientation sequence segment and the gaze point spatial coordinate sequence segment within the overlapping time period are marked as high attention behavior feedback segments, and the head rotation orientation sequence segment and the gaze point spatial coordinate sequence segment outside the overlapping time period are marked as low attention behavior feedback segments.

[0073] The high-attention behavior feedback segments are subjected to scene object reverse mapping processing. Based on the spatial coordinates of the gaze point in the sequence of gaze point spatial coordinates, the virtual object identifiers in the virtual reality test scene are queried to obtain a set of high-attention virtual object identifiers.

[0074] For each high-attention virtual object identifier in the set of high-attention virtual object identifiers, the attention duration is accumulated, and the cumulative gaze duration corresponding to each high-attention virtual object identifier is calculated;

[0075] The set of high-attention virtual object identifiers is sorted according to the cumulative gaze duration to generate a virtual object attention priority sequence arranged in descending order of cumulative gaze duration;

[0076] The scene region affiliation analysis is performed on the preceding virtual object identifier in the virtual object attention priority sequence to determine the scene functional region category to which the preceding virtual object identifier belongs in the virtual reality test scene.

[0077] A scene region attention behavior association mapping table is constructed based on the scene functional region category and the head micro-motion feature description in the gaze persistence mode description unit. The scene region attention behavior association mapping table records the typical head micro-motion feature description corresponding to each scene functional region category.

[0078] The high-attention behavior feedback segments, the low-attention behavior feedback segments, the virtual object attention priority sequence, and the scene area attention behavior association mapping table are encapsulated and processed to generate the test scene behavior feedback feature set with behavior feedback pattern labels.

[0079] The test scenario behavior feedback feature set with behavior feedback pattern label is encoded into a data transmission format that can be parsed by the test analysis terminal, and the encoded test scenario behavior feedback feature set with behavior feedback pattern label is transmitted to the test analysis terminal through a preset communication protocol interface.

[0080] In this embodiment of the invention, for example, after obtaining the behavioral feedback response pattern description set, the server performs attention focus annotation processing on the raw behavioral data stream uploaded by the head-mounted display device. The server first reads the head micro-motion feature description and the gaze stability feature description from the gaze persistence pattern description unit and performs fusion processing. Specifically, in a virtual assembly test scenario, when the user continuously observes the torque prompt panel, the head yaw and pitch angles exhibit low-amplitude, stable-frequency micro-motions, and the gaze placement point forms a small drift envelope area within the prompt text area. The server combines the dominant frequency of the head micro-motions, the energy amplitude, and the gaze drift envelope area into a gaze attention focus indication vector, used to characterize that the user has clear attention to the prompt panel during this time period.

[0081] The server then reads the dominant rhythm cycle from the control rhythm pattern description unit and aligns this dominant rhythm cycle with the gaze focus indicator vector in time. Specifically, while the user gazes at the target nut hole, the handheld control device uploads a stable sequence of alternating pressing rising edges and relaxing falling edges. The server maps this control rhythm event and the gaze persistence event to the same time axis and determines the time period when they overlap. This overlapping time period indicates that the user is not only looking at the target area but also simultaneously performing effective control. Therefore, the server marks the head rotation orientation sequence segment and the gaze placement spatial coordinate sequence segment within this time period as high-attention behavioral feedback segments. For time periods when the user briefly scans the edge of the tool rack without forming a stable gaze and without accompanying regular control, the server marks the corresponding head rotation orientation sequence segment and gaze placement spatial coordinate sequence segment as low-attention behavioral feedback segments.

[0082] In this embodiment of the invention, high-attention behavioral feedback segments and low-attention behavioral feedback segments can be determined using an attention score, Sfocus. The server calculates the attention score Sfocus based on gaze persistence duration (Tdwell), drift envelope area (A), head micromotion energy amplitude (E), and the temporal overlap (Oevent) between gaze persistence events and manipulation rhythm events. ,in, and These are the weighting coefficients, and Norm represents normalization, and ε is a constant to prevent the denominator from being zero. When Sfocus is not less than the preset high attention threshold θfocus, the server marks the corresponding behavioral feedback segment as a high attention behavioral feedback segment; when Sfocus is less than θfocus, the server marks the corresponding behavioral feedback segment as a low attention behavioral feedback segment. Thus, the server can comprehensively determine the user's level of attention to objects or areas in the virtual reality test scene based on gaze persistence, gaze stability, head micro-movement characteristics, and control synchronization.

[0083] The server continues to perform scene object reverse mapping processing on high-attention behavior feedback segments. Based on the 3D coordinates in the spatial coordinate sequence of the gaze point, the server queries the object index table of the virtual reality test scene, converting the virtual model, interface control, or prompt area where the coordinates fall into the sequence into the corresponding virtual object identifier. For example, when the gaze point continuously falls on the target nut, virtual wrench, red warning box, and torque value panel, the server obtains the identifiers of these objects respectively, forming a set of high-attention virtual object identifiers. The server then performs attention duration accumulation processing on each object identifier in the set, summing the gaze dwell time belonging to the same object in multiple high-attention segments to obtain the cumulative gaze dwell time corresponding to each object.

[0084] In this embodiment of the invention, the scene object reverse mapping process is performed based on the object space index table of the virtual reality test scene. The server pre-establishes a virtual object space index table Index={object_id,bounding_volume,region_type}, where object_id represents the virtual object identifier, bounding_volume represents the bounding box, bounding sphere, or collider region of the virtual object in the coordinate system of the virtual reality test scene, and region_type represents the scene functional region category to which the virtual object belongs. For the spatial coordinates gi of the gaze point in any high-attention behavior feedback segment, the server determines whether gi falls within the bounding_volume of any virtual object in the virtual object space index table. If gi falls within the bounding_volume of a virtual object, the object_id of that virtual object is determined as the corresponding high-attention virtual object identifier; if gi falls within the bounding_volume of multiple virtual objects simultaneously, the server selects the virtual object with the closest distance as the mapping result based on the distance between the user's viewpoint and the center point of each candidate virtual object. The test scenario behavior feedback feature set with behavior feedback pattern labels includes segment identifier (segment_id), start time (start_time), end time (end_time), focus level (focus_label), head micro-motion dominant frequency (fmain), head micro-motion energy amplitude (E), drift envelope area (A), dominant rhythm period (Tmain), virtual object identifier (object_id), scene functional region category (region_type), cumulative gaze dwell time (total_dwell_time), and scene region attention behavior association mapping table (map_table). After receiving the test scenario behavior feedback feature set, the test analysis terminal determines the virtual object to be optimized and its corresponding scene functional region based on focus_label, object_id, and region_type.

[0085] The server sorts the set of high-attention virtual object identifiers in descending order according to the cumulative gaze duration, generating a virtual object attention priority sequence. For example, the target nut has the longest cumulative gaze time, followed by the torque indicator panel, and then the red warning box. Based on this, the server determines the object hierarchy that the user focuses on in this test. The server further performs scene area affiliation analysis on the preceding objects in the virtual object attention priority sequence to determine whether they belong to the assembly operation area, tool selection area, parameter indicator area, or anomaly warning area. The server then combines these scene functional area categories with the head micro-movement feature descriptions in the gaze persistence mode description unit to construct a scene area attention behavior association mapping table. This mapping table records the typical head micro-movement states corresponding to each type of area. For example, the user exhibits low-frequency stable micro-movements in the parameter indicator area, high-energy amplitude micro-movements in the anomaly warning area, and continuous small-amplitude following rotations in the assembly operation area.

[0086] Finally, the server encapsulates high-attention behavior feedback segments, low-attention behavior feedback segments, virtual object attention priority sequences, and scene area attention behavior association mapping tables to generate a test scene behavior feedback feature set with behavior feedback pattern labels. The server encodes this feature set into a data transmission format that the test analysis terminal can parse, retaining segment timestamps, object identifiers, attention level labels, cumulative gaze duration, and area behavior mapping relationships in the encoded data, and sends it to the test analysis terminal through a preset communication protocol interface. Upon receiving the data, the test analysis terminal can directly identify the user's key attention areas, low attention areas, and objects with difficult interaction, and accordingly trigger virtual scene layout adjustments, enhanced prompts, and optimized interaction parameters.

[0087] In this embodiment of the invention, the method further includes:

[0088] The head rotation orientation sequence in the raw behavioral data stream transmitted by the head-mounted display device is obtained, and the head rotation orientation sequence is spatially encoded to map the continuous head rotation orientation sequence into a discrete spatial orientation interval identifier sequence.

[0089] Obtain the spatial coordinate sequence of gaze points belonging to the high-attention behavior feedback segment in the test scene behavior feedback feature set with the behavior feedback mode label, and perform spatiotemporal synchronization binding processing on the spatial coordinate sequence of gaze points and the spatial orientation interval identifier sequence to generate a gaze orientation synchronization pairing sequence.

[0090] Extract the corresponding spatial orientation interval identifier for each line-of-sight orientation synchronization pairing node in the line-of-sight orientation synchronization pairing sequence, and count the frequency of each spatial orientation interval identifier in the line-of-sight orientation synchronization pairing sequence;

[0091] A histogram of user head orientation preference distribution is constructed based on the frequency of occurrence of each spatial orientation interval identifier in the line-of-sight orientation synchronization pairing sequence. The histogram of user head orientation preference distribution represents the distribution tendency of user head orientation in virtual reality test scenarios.

[0092] Extract the spatial orientation interval identifiers with the highest frequency from the user's head orientation preference distribution histogram, and use the spatial orientation interval identifiers with the highest frequency as the user's dominant orientation preference interval;

[0093] Based on the user-dominant orientation preference range, the orientation adaptation evaluation process is performed on the layout of virtual objects in the virtual reality test scene, and the orientation deviation angle between each virtual object in the virtual reality test scene and the user-dominant orientation preference range is calculated.

[0094] Based on the aforementioned orientation deviation angle, the layout of virtual objects in the virtual reality test scene is adjusted adaptively, and virtual objects whose orientation deviation angle from the user's dominant orientation preference range exceeds a preset deviation threshold are repositioned within the user's dominant orientation preference range.

[0095] The repositioned virtual object is subjected to view distance optimization processing. The display size scaling factor of the virtual object is calculated based on the distance between the repositioned virtual object and the user's viewpoint in the virtual reality test scene. The rendering display size of the virtual object is adjusted according to the display size scaling factor.

[0096] The virtual object position parameters after layout adaptive adjustment and the virtual object display size scaling factor after view distance optimization are encapsulated into a scene layout optimization parameter set, and the scene layout optimization parameter set is sent to the scene rendering engine.

[0097] Obtain user behavior feedback verification data corresponding to the scene layout optimization parameter set, perform gaze duration comparison analysis on the gaze persistence mode description unit in the user behavior feedback verification data, and generate a scene layout optimization effect verification report.

[0098] In this embodiment of the invention, the method further includes:

[0099] Extract the cross-channel behavior response co-occurrence matrix from the multi-source behavior response synchronization feature set, and normalize the co-occurrence intensity of the cross-channel behavior response co-occurrence matrix to obtain the normalized cross-channel behavior response co-occurrence matrix;

[0100] The normalized cross-channel behavioral response co-occurrence matrix is ​​subjected to threshold filtering to extract head-eye-hand cross-channel ternary coupling patterns whose normalized co-occurrence intensity exceeds a preset co-occurrence intensity threshold. The head-eye-hand cross-channel ternary coupling pattern represents a composite behavioral response pattern in which head rotation behavior, gaze behavior, and hand manipulation behavior occur simultaneously.

[0101] Obtain the head rotation primitive segment, gaze persistence primitive segment, and force gradient primitive segment corresponding to the head-eye-hand cross-channel ternary coupling mode. Take the intersection of the time intervals of the head rotation primitive segment, gaze persistence primitive segment, and force gradient primitive segment to obtain the ternary coupling time window.

[0102] Within the three-element coupling time window, the control trajectory primitive fragments of the controller displacement trajectory sequence are extracted, and the control trajectory primitive fragments are classified into linear motion control trajectory primitives, arc motion control trajectory primitives, and polygonal motion control trajectory primitives.

[0103] The kinematic parameters of the linear motion control trajectory primitive, the arc motion control trajectory primitive, and the polyline motion control trajectory primitive are calculated and processed respectively. The motion direction vector and motion speed sequence of the linear motion control trajectory primitive are extracted, the curvature radius sequence and angular velocity sequence of the arc motion control trajectory primitive are extracted, and the turning point coordinate sequence and turning angle sequence of the polyline motion control trajectory primitive are extracted.

[0104] The gaze target virtual object is determined based on the gaze dwelling primitive fragment contained in the head-eye-hand cross-channel three-element coupling mode. The spatial position coordinates of the gaze target virtual object are compared with the motion direction vector of the linear motion control trajectory primitive to calculate the angular deviation between the gaze target direction and the control motion direction.

[0105] The angular deviation, the motion rate sequence, the radius of curvature sequence, the angular velocity sequence, the turning point coordinate sequence, and the turning angle sequence are concatenated to generate a head-eye-hand ternary coupled control behavior feature vector.

[0106] Cluster analysis is performed on the head-eye-hand ternary coupled manipulation behavior feature vectors corresponding to multiple ternary coupled time windows, and the head-eye-hand ternary coupled manipulation behavior feature vectors are aggregated into different manipulation behavior modal clusters. Each manipulation behavior modal cluster corresponds to a typical head-eye-hand coordinated manipulation behavior modality.

[0107] Extract the cluster center feature vector of each control behavior mode cluster, and use the cluster center feature vector as the typical control behavior mode feature description of that control behavior mode;

[0108] User control behavior modality fingerprints are constructed based on the proportion of occurrence of each typical control behavior modality feature description in the user group. The user control behavior modality fingerprints are then bound and stored with user identifiers to generate a user control behavior modality fingerprint database.

[0109] The user control behavior modal fingerprint database is input into the adaptive adjustment module of the virtual reality test scene. The adaptive adjustment module performs personalized adaptation processing on the interaction mode and control sensitivity parameters in the virtual reality test scene according to the current user's user control behavior modal fingerprint.

[0110] In this embodiment of the invention, for example, after obtaining a set of multi-source behavioral response synchronization features, the server extracts a cross-channel behavioral response co-occurrence matrix from it and normalizes the co-occurrence intensity values ​​in the matrix, so that head rotation, eye gaze, and hand manipulation form a unified and comparable intensity expression under different sampling frequencies and different dimensions. Specifically, in a virtual assembly test scenario, when the user looks at the target nut, simultaneously fine-tunes their head, and simultaneously moves towards the nut and presses the grasp button, the server records a high co-occurrence intensity of this group of behaviors in the co-occurrence matrix; when the user only briefly scans the tool rack and does not perform a pressing operation, the server records a lower co-occurrence intensity. The server performs threshold screening on the normalized cross-channel behavioral response co-occurrence matrix and extracts head-eye-hand cross-channel ternary coupling patterns with normalized co-occurrence intensities exceeding a preset threshold, which are used to characterize composite behavioral response patterns in which head rotation, eye gaze, and hand manipulation occur simultaneously.

[0111] The server then acquires the head rotation primitive segment, gaze persistence primitive segment, and force gradation primitive segment corresponding to the ternary coupling mode, and performs intersection processing on the time intervals of the three types of segments to obtain the ternary coupling time window. For example, if the user's head is facing the assembly table between 12.4 seconds and 13.8 seconds, and their gaze is fixed on the target nut, while the controller trigger button is gradually pressed, the server determines the time period covered by all three as the effective ternary coupling time window. Within this window, the server extracts the control trajectory primitive segment from the controller displacement trajectory sequence and classifies it according to the spatial trajectory shape. The trajectory of the controller moving directly from the user's hand to the nut position is classified as a linear motion control trajectory primitive; the trajectory of rotating and tightening around the nut is classified as an arc motion control trajectory primitive; and the trajectory of avoiding the virtual baffle and then approaching the target is classified as a polygonal motion control trajectory primitive.

[0112] The server further calculates kinematic parameters for different types of control trajectory primitives. For linear motion control trajectory primitives, the server extracts the motion direction vector and motion velocity sequence; for arc motion control trajectory primitives, the server extracts the radius of curvature sequence and angular velocity sequence; for polygonal motion control trajectory primitives, the server extracts the inflection point coordinate sequence and inflection angle sequence. Based on the gaze-holding primitive fragment in the three-element coupling mode, the server determines the gaze target virtual object, such as a target nut or virtual wrench, and compares the spatial coordinates of this object with the motion direction vector of the linear motion control trajectory primitive to calculate the angular deviation between the gaze target direction and the control motion direction. A smaller angular deviation indicates that the user's hand operation is accurately pointing at the gaze target, while a larger angular deviation indicates that although the user is looking at the target, the control direction has deviated.

[0113] The server concatenates features from angular deviation, motion rate sequence, radius of curvature sequence, angular velocity sequence, turning point coordinate sequence, and turning angle sequence to generate a head-eye-hand three-dimensional coupled control behavior feature vector. The server performs cluster analysis on the feature vectors corresponding to multiple three-dimensional coupled time windows, aggregating them into different control behavior modal clusters. For example, samples of stable straight-line approach to the target form the precise grasping modal cluster, samples of low-speed arc rotation form the fine adjustment modal cluster, and samples of frequent broken-line correction form the hesitation correction modal cluster. The server extracts the cluster center feature vector of each control behavior modal cluster as a typical control behavior modal feature description and constructs a user control behavior modal fingerprint based on the proportion of each modality in the user group. The server binds and stores this fingerprint with the user identifier, generating a user control behavior modal fingerprint database, which is then input into the adaptive adjustment module of the virtual reality test scene. This module adjusts the interaction method and control sensitivity parameters according to the current user's modal fingerprint. For example, it reduces the controller movement sensitivity and enhances target attraction assistance for users with a high proportion of hesitation correction modality, and maintains a high response speed for users with a high proportion of precise grasping modality, thereby achieving personalized adaptation.

[0114] In this embodiment of the invention, the server normalizes the cross-channel behavior response co-occurrence matrix M to obtain a normalized cross-channel behavior response co-occurrence matrix. : .when When the intensity exceeds the preset co-occurrence threshold θco, the server identifies the corresponding head rotation primitive segment, gaze persistence primitive segment, force application gradient primitive segment, and control trajectory primitive segment as candidate head-eye-hand cross-channel ternary coupling modes. For the control trajectory primitive segment, the server classifies the spatial trajectory morphology based on the ratio ρ of the straight-line distance Dstraight between the beginning and end points of the trajectory to the actual trajectory length Lpath: ρ = Dstraight / Lpath. When ρ is not less than the preset straight-line ratio threshold θline, the server classifies the corresponding control trajectory primitive segment as a linear motion control trajectory primitive; when ρ is less than θline and the average curvature κmean is within the preset arc curvature range [κmin, κmax], the server classifies the corresponding control trajectory primitive segment as an arc motion control trajectory primitive; when the number of curvature abrupt change points in the trajectory is not less than the preset turning point number threshold Nth, the server classifies the corresponding control trajectory primitive segment as a polyline motion control trajectory primitive. The head-eye-hand ternary coupled control behavior feature vector Ftri is generated by concatenating the following fields: Ftri=[Δangle,mean(v),max(v),var(v),mean(κ),var(κ),mean(ωangular),Nturn,mean(φturn),var(φturn),Tdwell,E,A], where Δangle represents the angular deviation between the gaze target direction and the control movement direction, v represents the movement rate sequence, κ represents the curvature sequence, ωangular represents the angular velocity sequence, Nturn represents the number of turning points, φturn represents the turning angle sequence, Tdwell represents the gaze dwell time, E represents the amplitude of head micro-motion energy, and A represents the drift envelope area. When the server performs cluster analysis on multiple Ftri, it can use K-means clustering, DBSCAN density clustering, or hierarchical clustering. When using K-means clustering, the server sets the number of clusters K to an integer between 2 and 8, and selects the optimal K value using the silhouette coefficient. When using DBSCAN density clustering, the server determines the neighborhood radius eps based on the k-nearest neighbor distance curve, and determines the minimum number of samples min_samples based on the number of ternary coupled samples. After clustering, the server determines the center vector of each control behavior modality cluster as the typical control behavior modality feature description. The user control behavior modality fingerprint is generated according to the proportion of each typical control behavior modality in the user's behavior samples. Let the total number of ternary coupled samples corresponding to user u be Nu, and the number of occurrences of the k-th class of control behavior modality be Nuk, then the user's control behavior modality fingerprint Fingerprint(u) is: Fingerprint(u) = [Nu1 / Nu, Nu2 / Nu, ..., NuK / Nu]. The adaptive adjustment module adjusts the parameters according to Fingerprint(u).When the proportion of the hesitation correction mode is greater than the preset hesitation correction ratio threshold θhesitate, the adaptive adjustment module reduces the controller displacement sensitivity and increases the target adsorption intensity; when the proportion of the precise grasping mode is greater than the preset precise grasping ratio threshold θprecise, the adaptive adjustment module maintains or increases the controller response speed; when the proportion of the arc fine-tuning mode is greater than the preset fine-tuning ratio threshold θfine, the adaptive adjustment module reduces the rotation step angle and increases the angle correction accuracy.

[0115] In this embodiment of the invention, the method further includes:

[0116] Obtain high-attention behavioral feedback segments from the behavioral feedback feature set of the test scene with behavioral feedback pattern labels, and extract the gaze stability feature description from the gaze persistence pattern description unit corresponding to each high-attention behavioral feedback segment.

[0117] The drift envelope area in the description of line-of-sight stability features is subjected to time-series change analysis, and the drift envelope areas of continuous high-attention behavior feedback segments are connected to form a drift envelope area change curve.

[0118] The drift envelope area change curve is subjected to trend decomposition processing, which decomposes the drift envelope area change curve into monotonic trend components and periodic fluctuation components.

[0119] Based on the monotonic trend components, the long-term trend direction of gaze stability is determined, and the period when the long-term trend direction is that the drift envelope area continues to increase is marked as the period of visual fatigue accumulation.

[0120] Obtain the distribution density of low-attention behavioral feedback segments in the test scene behavioral feedback feature set with behavioral feedback pattern labels during the visual fatigue accumulation period, and calculate the proportion of low-attention behavioral feedback segments to all behavioral feedback segments during the visual fatigue accumulation period.

[0121] Obtain the dominant rhythm cycle in the control rhythm pattern description unit during the visual fatigue accumulation period, and calculate the offset ratio of the dominant rhythm cycle relative to the baseline dominant rhythm cycle.

[0122] A comprehensive assessment vector for visual fatigue is constructed based on the proportion of low-attention behavioral feedback segments to all behavioral feedback segments and the offset ratio of the dominant rhythm cycle relative to the baseline dominant rhythm cycle.

[0123] The visual fatigue level comprehensive evaluation vector is matched with the preset multi-level visual fatigue level standard vector. The visual fatigue level corresponding to the preset multi-level visual fatigue level standard vector with the highest similarity is used as the visual fatigue level judgment result of the current user.

[0124] Based on the visual fatigue level determination result, a visual stimulus intensity adjustment instruction is generated for the virtual reality test scene. The visual stimulus intensity adjustment instruction is used to reduce the brightness, contrast and movement speed of visual elements in the virtual reality test scene.

[0125] The visual fatigue level determination result and the start and end timestamps of the visual fatigue accumulation period are encapsulated into a visual fatigue monitoring record, and the visual fatigue monitoring record is appended to the user behavior feedback feature database.

[0126] In this embodiment of the invention, for example, after generating a test scenario behavioral feedback feature set with behavioral feedback pattern labels, the server continues to monitor the user's visual fatigue state during the test. The server first reads high-attention behavioral feedback segments from this feature set and extracts the gaze stability feature description from the gaze persistence pattern description unit corresponding to each high-attention behavioral feedback segment. Specifically, in the virtual assembly test scenario, when the user continuously observes the target nut, torque indicator panel, tool selection icon, and error warning box, the server extracts the gaze placement drift envelope area from these high-attention segments respectively. Initially, when the user gazes at the target nut, the gaze placement is concentrated near the nut hole, and the drift envelope area is small. After a period of testing, when the user views similar prompts, the gaze placement begins to expand its drift range between the prompt text, background edge, and adjacent icons, and the server records this increase in the drift envelope area.

[0127] The server performs time-series variation analysis on the drift envelope area in continuous high-attention behavior feedback segments, connecting the drift envelope areas of each segment chronologically to form a drift envelope area variation curve. This curve reflects the continuous change in the user's gaze stability during the test. The server further performs trend decomposition processing on this variation curve, decomposing it into a monotonic trend component and a periodic fluctuation component. The monotonic trend component reflects the long-term upward or downward state of the drift range, while the periodic fluctuation component reflects the local fluctuations caused by normal saccades, brief target searches, or scene switching. The server determines the long-term direction of gaze stability change based on the monotonic trend component and marks the time period in which the drift envelope area continuously increases as the visual fatigue accumulation period. For example, when the user observes the torque value panel in the latter half of the test, their gaze gradually becomes difficult to stabilize in the value area, and the drift envelope area continuously increases; the server identifies this period as the visual fatigue accumulation period.

[0128] The server then acquires the distribution density of low-attention behavioral feedback segments during the visual fatigue accumulation period and calculates the proportion of low-attention behavioral feedback segments to all behavioral feedback segments. Specifically, during the visual fatigue accumulation period, the user frequently and briefly scans the edges of the tool rack, prompt panel, and assembly table, but fails to establish stable fixation or effective control. The server counts the number and duration of these low-attention segments during this period. Simultaneously, the server acquires the dominant rhythmic cycle in the control rhythm pattern description unit during this visual fatigue accumulation period and compares it with the baseline dominant rhythmic cycle in the user's normal state, calculating the dominant rhythmic cycle offset ratio. For example, if the user's trigger presses become significantly slower during fatigue, and the interval between the rising edge of the press and the falling edge of the relaxation becomes longer, the server determines that a control rhythm offset has occurred.

[0129] The server constructs a comprehensive assessment vector for visual fatigue based on the proportion of low-attention behavioral feedback segments and the shift ratio of the dominant rhythm cycle. This vector reflects both decreased user attention stability and abnormal control rhythm. The server performs similarity matching between this comprehensive assessment vector and a preset multi-level visual fatigue standard vector, and determines the level corresponding to the standard vector with the highest similarity as the current user's visual fatigue level. For example, when the proportion of low-attention segments is high and the dominant rhythm cycle is significantly prolonged, the server determines that the user is in a medium-to-high level of visual fatigue. Based on this determination, the server generates visual stimulus intensity adjustment instructions for the virtual reality test scene to reduce the brightness, contrast, and movement speed of warning boxes, dynamic arrows, flashing borders, and highlighted textures in the scene, thereby reducing the user's visual burden. Finally, the server encapsulates the visual fatigue level determination result and the start and end timestamps of the visual fatigue accumulation period into a visual fatigue monitoring record, and appends this record to the user behavior feedback feature database for subsequent personalized scene adjustment and test report generation.

[0130] In this embodiment of the invention, when the server performs trend decomposition on the drift envelope area change curve A(t), a moving average decomposition method can be used. Let the sliding window length be L, then the monotonic trend component Trend(t) is: Trend(t) = 1 / L·ΣA(ti), i = 0, 1, ..., L-1. The periodic fluctuation component Season(t) is: Season(t) = A(t) - Trend(t). When Trend(t) satisfies Trend(t2) - Trend(t1) > θtrend within a continuous time period, and the length of this continuous time period is not less than the preset fatigue duration threshold Tfatigue, the server determines this continuous time period as the visual fatigue accumulation period. The comprehensive visual fatigue assessment vector Vfatigue is constructed as follows: Vfatigue=[Rlow,ΔTpress,ΔA,Dlow], where Rlow represents the proportion of low-attention behavioral feedback segments to all behavioral feedback segments during the visual fatigue accumulation period, ΔTpress represents the offset ratio of the current dominant rhythm cycle relative to the baseline dominant rhythm cycle, ΔA represents the growth ratio of the drift envelope area relative to the baseline drift envelope area, and Dlow represents the temporal distribution density of low-attention behavioral feedback segments. The server pre-stores multi-level visual fatigue standard vectors V0, V1, V2, and V3, corresponding to no fatigue, mild fatigue, moderate fatigue, and severe fatigue, respectively. The server calculates the cosine similarity between Vfatigue and each standard vector: Sim(Vfatigue,Vk)=(Vfatigue·Vk) / (||Vfatigue||·||Vk||). The server determines the level corresponding to the standard vector with the highest similarity as the current user's visual fatigue level assessment result. When the visual fatigue level is mild, the server generates a first visual stimulus intensity adjustment command, reducing the brightness of the highlighted object by 5% to 10%; when the visual fatigue level is moderate, the server generates a second visual stimulus intensity adjustment command, reducing the brightness of the highlighted object by 10% to 20% and reducing the movement speed of the dynamic prompt elements; when the visual fatigue level is severe, the server generates a third visual stimulus intensity adjustment command, reducing the brightness of the highlighted object by 20% to 30%, reducing the flicker frequency, and triggering the test analysis terminal to output a rest prompt.

[0131] In this embodiment of the invention, the method further includes:

[0132] Obtain the set of head-eye synchronization coupling primitive pairs in the multi-source behavior response synchronization feature set, and extract the spatial coordinate sequence of the gaze landing point corresponding to the gaze dwell primitive fragment in each head-eye synchronization coupling primitive pair.

[0133] The spatial trajectory smoothing process is performed on the spatial coordinate sequence segment of the line of sight landing point to eliminate high-frequency jitter noise in the spatial coordinate sequence segment of the line of sight landing point and obtain a smooth line of sight trajectory.

[0134] The smooth gaze trajectory is subjected to gaze shift recognition processing to extract the saccade jump primitive segments during the transition from one gaze dwell primitive segment to the next gaze dwell primitive segment in the smooth gaze trajectory.

[0135] Obtain the spatial coordinates of the starting gaze point and the ending gaze point corresponding to the saccade jump primitive fragment, and calculate the saccade jump vector based on the spatial coordinates of the starting gaze point and the ending gaze point.

[0136] A virtual object spatial topology graph is constructed by arranging all virtual objects in the virtual reality test scene according to their spatial layout relationship. In the virtual object spatial topology graph, nodes represent virtual objects and edges represent the spatial adjacency relationship between virtual objects.

[0137] Map the saccade jump vector onto the virtual object space topology graph to determine the starting and ending virtual object nodes of the saccade jump;

[0138] The frequency of occurrence of the transition relationship between the starting virtual object node and the ending virtual object node corresponding to saccade jumps in all head-eye synchronization coupling primitive pairs is counted, and an attention transfer probability matrix between virtual objects is constructed.

[0139] The graph traversal analysis is performed on the attention transfer probability matrix between the virtual objects, and the attention transfer path search algorithm is used to search for the virtual object access sequence with the highest cumulative transfer probability on the virtual object space topology graph to obtain the dominant path of user attention transfer.

[0140] Extract the attribute labels of each virtual object node on the dominant path of user attention transfer, and analyze the semantic logical order of user attention transfer in the virtual reality test scene based on the attribute labels;

[0141] The path similarity of the user attention transfer dominant path and the expected attention transfer path preset by the virtual reality test scene designer is compared, and the node sequence overlap between the user attention transfer dominant path and the expected attention transfer path is calculated.

[0142] The effectiveness of virtual reality test scene guidance is evaluated based on the overlap of the node sequence. Deviations of virtual object nodes that deviate from the expected attention transfer path are extracted from the dominant path of user attention transfer. These deviations of virtual object nodes are marked as weak nodes in scene guidance, and scene guidance enhancement strategies are generated for these weak nodes.

[0143] In this embodiment of the invention, for example, after obtaining the set of multi-source behavioral response synchronization features, the server further analyzes the user's attention transfer path in the virtual reality testing scenario. The server first extracts the set of head-eye synchronization coupling primitive pairs and reads the spatial coordinate sequence of the gaze placement point corresponding to the gaze dwelling primitive segment in each head-eye synchronization coupling primitive pair. Specifically, in the virtual assembly testing scenario, the user first gazes at the target nut, then checks the virtual wrench, and subsequently turns to the torque prompt panel and error warning box. The server obtains the gaze placement point coordinate sequence within these gaze dwelling stages as the basic data for attention transfer analysis.

[0144] The server performs spatial trajectory smoothing on the spatial coordinate sequence of the gaze point, filtering out high-frequency jitter noise caused by blinking, slight device vibration, or pupil recognition errors during eye movement acquisition, resulting in a smooth gaze trajectory. Subsequently, the server performs gaze shift recognition processing on the smooth gaze trajectory, extracting saccade jump primitive segments during the transition from one gaze-resident primitive segment to the next. For example, when the user's gaze quickly moves from the target nut to the wrench icon on the left tool rack, the server identifies this rapid movement segment as a saccade jump primitive segment; similarly, when the user turns from the torque value panel to the red error warning box, the server extracts the corresponding saccade jump segment.

[0145] The server obtains the spatial coordinates of the starting and ending gaze points for each saccadic jump primitive segment and calculates the saccadic jump vector based on these coordinates. This vector records the spatial origin, direction, and distance of the user's attention. Simultaneously, the server constructs a virtual object spatial topology graph based on the spatial layout of all virtual objects in the virtual reality test scene. Target nuts, virtual wrenches, tool rack buttons, torque indicator panels, error warning boxes, and assembly completion confirmation buttons are all treated as graph nodes, with adjacent or operationally related objects forming graph edges. The server maps the saccadic jump vector onto this topology graph, determining the starting and ending virtual object nodes for each saccadic jump.

[0146] The server statistically analyzes the frequency of transitions from the start node to the end node in all head-eye synchronization coupling primitive pairs, constructing an attention transition probability matrix between virtual objects. For example, if a user frequently transitions from the target nut to the virtual wrench, the transition probability is high; similarly, if the number of times a user mistakenly transitions from the tool rack to an unrelated decoration increases, this abnormal path is also recorded. The server performs graph traversal analysis on the attention transition probability matrix and employs an attention transition path search algorithm to search for the virtual object access sequence with the highest cumulative transition probability in the virtual object space topology graph, obtaining the user's dominant attention transition path. This dominant path reflects the user's typical observation sequence during the test, such as "target nut—virtual wrench—torque indicator panel—confirm button".

[0147] In this embodiment of the invention, assuming there are m virtual object nodes in the virtual object space topology graph, the number of times the user's gaze jumps from node i to node j is . Then the elements in the attention transfer probability matrix P between virtual objects for: The server identifies the node sequence with the highest cumulative transfer probability as the dominant path for user attention transfer, Path*. The path similarity Spath between the dominant path Path* and the desired attention transfer path Path0 is calculated based on the length of the longest common subsequence of the node sequences: Spath = LCS(Path*, Path0) / max(len(Path*), len(Path0)). When Spath is lower than a preset path similarity threshold θpath, the server marks virtual object nodes in Path* that do not appear at the corresponding position in Path0 as deviated virtual object nodes, and identifies these deviated virtual object nodes as weak nodes in scene guidance. The server generates scene guidance enhancement strategies based on the spatial location of the weak nodes, the category of scene functional areas, and the direction of user attention transfer. These scene guidance enhancement strategies include increasing the brightness of target prompts near the deviated nodes, adding directional guidance arrows, adjusting the position of the prompt panel, or reducing the display intensity of surrounding interfering objects.

[0148] The server further extracts attribute labels from each virtual object node on the dominant path of user attention transfer and analyzes its semantic logical order. The target nut is an assembly object, the virtual wrench is an operating tool, the torque indicator panel is a parameter prompt, and the confirmation button is a process submission. Based on this, the server determines whether the user's attention migrates in a reasonable order of "identifying the target—selecting the tool—reading parameters—completing confirmation." Subsequently, the server compares the path similarity of this dominant path with the expected attention transfer path preset by the scene designer, calculating the overlap of their node sequences. When the user's dominant path frequently skips the torque indicator panel and directly enters the confirmation button, the server determines that the guidance effectiveness of this key prompt node is insufficient. The server evaluates the overall guidance effect of the virtual reality test scene based on the node sequence overlap and marks virtual object nodes that deviate from the expected path as weak nodes in scene guidance. For these nodes, scene guidance enhancement strategies are generated, such as increasing the brightness of the torque indicator panel, adding connecting arrows, extending the prompt display time, or weakening surrounding distracting objects, to improve the consistency between the user's attention transfer path and the expected interaction process.

[0149] In this embodiment of the invention, the method further includes:

[0150] Obtain the set of manual control synchronization coupling primitive pairs in the multi-source behavior response synchronization feature set, perform pressure impact analysis on the force application gradient primitive fragment in each manual control synchronization coupling primitive pair, and extract the rate at which the button trigger intensity in the force application gradient primitive fragment rises from the baseline value to the peak value as the pressure impact feature.

[0151] Obtain the control trajectory primitive segment in each manual control synchronous coupling primitive pair, perform trajectory jitter analysis on the control trajectory primitive segment, and extract the instantaneous deviation amplitude sequence of the controller displacement trajectory relative to the smooth reference trajectory in the control trajectory primitive segment as the trajectory jitter degree feature.

[0152] The pressing impact feature and the trajectory jitter feature are time-aligned and combined to form a control stability feature binary.

[0153] All control stability feature pairs obtained in the current virtual reality test scenario are used to form a control stability feature sample set. Outlier detection processing is performed on the control stability feature sample set to identify outlier control stability feature pairs that deviate from the main distribution area of ​​the sample.

[0154] Extract the time interval of the force-gradient primitive segment corresponding to the outlier control stability feature binary and mark the time interval as the control anomaly micro-period.

[0155] The head rotation orientation sequence segment and the gaze point spatial coordinate sequence segment are retrieved from the original behavioral data stream transmitted by the head-mounted display device during the micro-period of the control abnormality. The head rotation orientation sequence segment is subjected to abrupt change detection processing to extract the abrupt change event of the head rotation direction.

[0156] The gaze interruption detection process is performed on the spatial coordinate sequence segment of the gaze point, and the transient jump event of the spatial coordinate of the gaze point is extracted. The transient jump event represents the user's gaze suddenly jumping from the original gaze target to a new gaze target within a micro-period of abnormal operation.

[0157] The timestamp of the sudden change in head rotation direction and the timestamp of the transient jump in the spatial coordinates of the gaze point are analyzed for synchronization. The time offset between the sudden change in head rotation direction and the transient jump in the spatial coordinates of the gaze point is calculated. The sudden change in head rotation direction and the transient jump in the spatial coordinates of the gaze point with a time offset less than a preset synchronization threshold are marked as head-eye stress coordination events.

[0158] Obtain the spatial coordinates of the termination point of the gaze landing point in the transient jump event of the head-eye stress coordination event, and query the scene stimulus source object in the corresponding virtual reality test scene based on the spatial coordinates of the termination point of the gaze landing point. Mark the scene stimulus source object as the interference stimulus source that causes abnormal control.

[0159] The virtual object identifier of the interference stimulus, the time interval of the micro-period of the control abnormality, and the control stability feature tuple are encapsulated into a control abnormality event record, and the control abnormality event record is written into the interference source log database of the virtual reality test scene.

[0160] In this embodiment of the invention, for example, after obtaining the multi-source behavioral response synchronization feature set, the server continues to identify anomalies in the stability of the user's hand operation. The server first extracts a set of hand-controlled synchronization coupling primitive pairs from the multi-source behavioral response synchronization feature set and reads the force-gradient primitive fragment in each hand-controlled synchronization coupling primitive pair. Specifically, in a virtual assembly test scenario, when the user presses the controller trigger button to grab the virtual wrench, the button trigger intensity gradually rises from the baseline value to the peak value; when the user is disturbed by a sudden flashing warning, the button trigger intensity will rise sharply. The server calculates the rate at which this intensity rises from the baseline value to the peak value and uses this rate as the pressing impact characteristic. The greater the pressing impact, the more rapid the user's pressing action, and the more likely there is tension, accidental touch, or influence from external stimuli.

[0161] The server simultaneously acquires the control trajectory primitive segments from each manual control synchronous coupling primitive pair and performs trajectory jitter analysis on the controller displacement trajectory. The server first generates a smooth reference trajectory based on the controller's continuous displacement points, then calculates the instantaneous deviation sequence of the actual trajectory relative to the smooth reference trajectory, using this as a trajectory jitter characteristic. For example, when the user steadily moves the virtual wrench towards the target nut, the controller trajectory is basically consistent with the smooth reference trajectory; when a red error warning box suddenly pops up in the scene, the user's hand experiences a brief tremor, the actual trajectory deviates from the reference trajectory, and the server records a significantly increased trajectory jitter.

[0162] The server performs time-aligned and combined processing of the pressure impact and trajectory jitter features within the same hand-controlled synchronous coupling primitive pair to form a control stability feature tuple. This tuple simultaneously expresses whether the user's button application force is rapid and whether the hand's spatial movement is stable. The server constructs a control stability feature sample set from all control stability feature tuples in the current virtual reality test scene and performs outlier detection processing. Normal grasping, rotating, and confirming operations form a stable sample main distribution area, while samples with significantly higher pressure impact and synchronously increased trajectory jitter are identified by the server as outlier control stability feature tuples.

[0163] The server further extracts the time interval of the force-gradient primitive segment corresponding to the outlier control stability feature binary and marks this time interval as a micro-period of control anomaly. Subsequently, the server backtracks to obtain the head rotation orientation sequence segment and the gaze placement spatial coordinate sequence segment uploaded by the head-mounted display device within this micro-period. The server performs abrupt change detection on the head rotation orientation sequence segment to identify the event of a sudden change in the head rotation direction of the user's head from the assembly table direction to the right prompt area; at the same time, it performs gaze interruption detection on the gaze placement spatial coordinate sequence segment to identify the event of a transient jump in the user's gaze from the original nut hole position to a new target.

[0164] The server performs synchronization analysis on the timestamps of head rotation direction abrupt changes and gaze jump events, calculating the time offset between them. When the time offset is less than a preset synchronization threshold, the server marks the combination of head abrupt change and gaze jump as a head-eye stress coordination event. Next, the server reads the spatial coordinates of the termination gaze point of the gaze jump event within this head-eye stress coordination event and queries the virtual reality test scene object index table to determine the scene stimulus object corresponding to those coordinates. For example, if the termination gaze point falls on a suddenly flashing red warning box, the server marks that warning box as an interfering stimulus that caused the control anomaly. Finally, the server encapsulates the virtual object identifier of the interfering stimulus, the start and end times of the control anomaly micro-period, and the corresponding control stability feature tuple into a control anomaly event record and writes it to the interference source log database of the virtual reality test scene. This record is used for subsequent reduction of interfering stimulus intensity, adjustment of warning appearance location, and optimization of anomaly prompting methods.

[0165] In this embodiment of the invention, the method further includes:

[0166] Extract all head-eye synchronization coupling primitive pairs from the multi-source behavior response synchronization feature set, perform rotation amplitude discretization processing on the head rotation primitive segment in each head-eye synchronization coupling primitive pair, and quantize the rotation angle of the head rotation primitive segment into a standard rotation amplitude range identifier.

[0167] For each head-eye synchronization coupling primitive pair, the gaze persistence primitive segment is discretized, and the gaze persistence primitive segment is mapped to the gaze function area identifier according to the virtual reality test scene function partition to which the spatial coordinates of the gaze landing point corresponding to the gaze persistence primitive segment belong.

[0168] The standard rotation amplitude range identifier and the gaze function area identifier are combined into a head-eye coordination state binary symbol. The head-eye coordination state binary symbols corresponding to consecutive head-eye synchronization coupling primitive pairs are concatenated into a head-eye coordination state symbol sequence according to time order.

[0169] The head-eye coordination state symbol sequence is segmented using a sliding time window of preset length, and each segment of the head-eye coordination state symbol sequence within the sliding time window is taken as a head-eye coordination behavior pattern unit.

[0170] Frequent pattern mining is performed on all head-eye coordination behavior pattern units. A sequence pattern mining algorithm is used to extract frequent head-eye coordination behavior pattern subsequences whose frequency of occurrence exceeds a preset support threshold. The frequent head-eye coordination behavior pattern subsequences represent the head-eye coordination behavior habit patterns that users repeatedly exhibit in virtual reality test scenarios.

[0171] For each frequently obtained head-eye coordinated behavior pattern subsequence, a confidence index is calculated, which is the probability of the frequently obtained head-eye coordinated behavior pattern subsequence appearing in all head-eye coordinated behavior pattern units.

[0172] After sorting the confidence index from high to low, extract the preceding frequent head-eye coordination behavior pattern subsequence, restore the standard rotation amplitude range identifier in the preceding frequent head-eye coordination behavior pattern subsequence to the rotation angle range of the head rotation primitive segment, restore the gaze function area identifier in the preceding frequent head-eye coordination behavior pattern subsequence to the spatial range of the virtual reality test scene function partition, and generate the restored frequent head-eye coordination behavior pattern description.

[0173] The restored frequent head-eye coordination behavior pattern is described as a user head-eye coordination behavior habit model. The user head-eye coordination behavior habit model is applied to the predictive rendering process of the virtual reality test scene. Based on the prefix of the symbol sequence of the currently occurring head-eye coordination state, the user will be able to look at the functional partition of the virtual reality test scene. The virtual objects in the predicted functional partition of the virtual reality test scene are preloaded with rendering resources.

[0174] Obtain the virtual reality test scene running frame rate data after rendering resource preloading processing, compare the running frame rate data with the historical running frame rate data without using predictive rendering processing to generate a predictive rendering effect evaluation record.

[0175] The user's head-eye coordination behavior habit model is bound and stored with the user identifier to build a user head-eye coordination behavior habit model library, which can be used for personalized rendering scheduling and calling in subsequent virtual reality test scene experiences.

[0176] In this embodiment of the invention, for example, after obtaining the set of multi-source behavioral response synchronization features, the server further constructs a user head-eye coordinated behavior habit model. The server first extracts all head-eye synchronization coupling primitive pairs and discretizes the rotation amplitude of the head rotation primitive segment in each pair. Specifically, in a virtual assembly test scenario, when the user slightly turns from the assembly table to the torque indicator panel, the server quantifies the head rotation angle as a small-amplitude rotation range; when the user clearly turns from the assembly area directly in front to the left tool rack, the server quantifies it as a medium-amplitude rotation range; when the user quickly turns to the right error warning box, the server quantifies it as a large-amplitude rotation range and forms a standard rotation amplitude range identifier.

[0177] The server then discretizes the gaze persistence primitive segment within each head-eye synchronization coupling primitive pair. Based on the virtual reality test scene functional partition to which the spatial coordinates of the gaze point in the gaze persistence primitive segment belong, the server maps it to a gaze function region identifier. For example, when the gaze point is located in the area containing nuts, bolts, and mounting holes, the server maps it to an assembly operation area identifier; when the gaze point is located in the area containing tool icons such as wrenches and screwdrivers, the server maps it to a tool selection area identifier; when the gaze point is located in the area containing torque values, step descriptions, and error warning boxes, the server maps it to a parameter prompt area, process description area, or abnormal warning area identifier, respectively. The server combines the standard rotation amplitude range identifier with the gaze function region identifier into a head-eye coordination state binary symbol, and concatenates the binary symbols corresponding to consecutive head-eye synchronization coupling primitive pairs into a head-eye coordination state symbol sequence according to time order.

[0178] The server segments the head-eye coordination state symbol sequence using a preset-length sliding time window, treating each segment as a head-eye coordination behavior pattern unit. For example, a window containing the symbol combination "small rotation—assembly operation area," "medium rotation—tool selection area," and "small return rotation—assembly operation area" indicates that the user exhibits a typical back-and-forth observation behavior between the assembly target and tool selection. The server performs frequent pattern mining on all head-eye coordination behavior pattern units, using a sequence pattern mining algorithm to extract frequent head-eye coordination behavior pattern subsequences that occur more frequently than a preset support threshold. These subsequences characterize recurring head-eye coordination behavior habits in the test scenario, such as a stable behavior pattern of "first looking at the assembly area, then at the tool area, and then back to the assembly area."

[0179] The server further calculates the probability of each frequent head-eye coordination behavior pattern subsequence appearing in all head-eye coordination behavior pattern units, and uses this probability as a confidence index. The server sorts the subsequences by confidence from high to low, extracts preceding frequent pattern subsequences, and restores the standard rotation amplitude range markers to specific head rotation angle ranges, and restores the gaze function area markers to the spatial range of functional partitions in the virtual reality test scene, generating a restored description of the frequent head-eye coordination behavior patterns. For example, the server restores "medium amplitude rotation—tool selection area" to "head turning approximately 20 to 40 degrees to the left and gazing at the tool rack spatial range," making the behavior habit model interpretable.

[0180] The server uses the reconstructed frequent head-eye coordination behavior patterns as a user head-eye coordination behavior habit model and applies this model to predictive rendering. Specifically, when the server detects that the current symbol sequence prefix is ​​"assembly operation area - tool selection area", it predicts that the user is about to look back at the assembly operation area based on historical high-confidence patterns and notifies the rendering engine in advance to preload the nut, wrench contact point, highlight outline, and tooltip texture resources within that area. The server obtains the preloaded scene running frame rate data and compares it with historical frame rate data that did not use predictive rendering processing to generate a predictive rendering effect evaluation record. Finally, the server binds and stores the user head-eye coordination behavior habit model with the user identifier, constructing a user head-eye coordination behavior habit model library, which can be used for subsequent virtual reality test scenarios to perform personalized rendering scheduling according to different users' observation habits.

[0181] In this embodiment of the invention, the method further includes:

[0182] Obtain the gaze persistence mode description unit from the behavior feedback response mode description set, extract the head micro-motion frequency domain energy distribution from the gaze persistence mode description unit, perform frequency band division processing on the head micro-motion frequency domain energy distribution, and divide the head micro-motion frequency domain energy distribution along the frequency axis into low-frequency micro-motion energy component, mid-frequency micro-motion energy component and high-frequency micro-motion energy component;

[0183] Obtain the control rhythm pattern description unit from the behavior feedback response pattern description set, extract the dominant rhythm cycle from the control rhythm pattern description unit, and convert the dominant rhythm cycle into a rhythm frequency value.

[0184] The low-frequency micro-motion energy component, the medium-frequency micro-motion energy component, the high-frequency micro-motion energy component and the rhythm frequency value are subjected to cross-dimensional feature fusion processing, and the four are spliced ​​together into a physiological behavior rhythm coupling feature vector.

[0185] Within the complete experience cycle of the virtual reality test scenario, physiological behavior rhythm coupling feature vectors of multiple time periods are obtained in chronological order to construct a time series of physiological behavior rhythm coupling feature vectors.

[0186] The time series of the coupled feature vector of physiological behavior rhythm is subjected to change point detection processing to locate the abrupt time points in the time series of the coupled feature vector of physiological behavior rhythm where the statistical characteristics change significantly, and the time period between adjacent abrupt time points is marked as the steady state stage of physiological behavior rhythm.

[0187] Calculate the mean vector of the physiological behavior rhythm coupling feature vector within each physiological behavior rhythm homeostasis phase, and use the mean vector as the rhythm homeostasis feature representative vector of the corresponding physiological behavior rhythm homeostasis phase;

[0188] The rhythm homeostasis feature representative vectors of adjacent physiological behavior rhythm homeostasis stages are subjected to differential quantification processing, and the vector distance between adjacent rhythm homeostasis feature representative vectors is calculated. The vector distance is used as the physiological behavior rhythm migration amplitude.

[0189] Extract the proportion of high-attention behavioral feedback segments from the multi-source behavioral response synchronization feature set within each physiological behavioral rhythm homeostasis phase, and calculate the proportion of high-attention behavioral feedback segments in each physiological behavioral rhythm homeostasis phase.

[0190] The correlation analysis was performed between the amplitude of the physiological behavior rhythm migration and the proportion of high-attention behavior feedback segments in each physiological behavior rhythm steady state stage, and a correlation mapping table between the amplitude of the physiological behavior rhythm migration and the proportion of high-attention behavior feedback segments was constructed.

[0191] Based on the association mapping table, identify the threshold range of physiological behavior rhythm migration amplitude that leads to a significant decrease in the proportion of high-attention behavior feedback segments, and mark the virtual reality test scene interaction event sequence corresponding to the physiological behavior rhythm migration amplitude threshold range as the key trigger event sequence for user experience state transition.

[0192] The scene interaction events in the key trigger event sequence of the user experience state transition are classified and summarized according to event type to generate a list of user experience state transition trigger events grouped by event type. The list of user experience state transition trigger events is then sent to the experience flow optimization module of the virtual reality test scene.

[0193] This invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 performs the aforementioned method for extracting user behavior feedback features in a virtual reality testing scenario. Figure 4 As shown, Figure 4 This is a structural block diagram of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a memory 111, a processor 112, and a communication unit 113. To enable data transmission or interaction, the memory 111, processor 112, and communication unit 113 are electrically connected to each other directly or indirectly. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0194] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the foregoing illustrative discussions are not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in accordance with the foregoing teachings. These embodiments were chosen and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the disclosure and to employ various embodiments with different modifications to suit a particular intended application.

Claims

1. A method for extracting user behavior feedback features in a virtual reality testing scenario, characterized in that, The method includes: Acquire raw behavioral data stream transmitted by a head-mounted display device, wherein the raw behavioral data stream transmitted by the head-mounted display device includes a timestamp-aligned sequence of head rotation orientation and a sequence of spatial coordinates of gaze points; Acquire the raw control data stream transmitted by the handheld control device, wherein the raw control data stream transmitted by the handheld control device includes a key trigger intensity sequence and a controller displacement trajectory sequence; A preset cross-modal behavior response fusion network is invoked to perform multi-source behavior response synchronization association processing on the head rotation orientation sequence, the gaze point spatial coordinate sequence, the button trigger intensity sequence, and the controller displacement trajectory sequence, generating a multi-source behavior response synchronization feature set. Based on the multi-source behavioral response synchronization feature set, user behavior feedback pattern mining processing is performed to generate a behavioral feedback response pattern description set, which includes gaze persistence pattern description unit and manipulation rhythm pattern description unit. The original behavioral data stream transmitted by the head-mounted display device is annotated with focus according to the behavioral feedback response pattern description set to generate a test scenario behavioral feedback feature set with behavioral feedback pattern labels, and the test scenario behavioral feedback feature set with behavioral feedback pattern labels is transmitted to the test analysis terminal to trigger the scenario optimization response. The method invokes a preset cross-modal behavior response fusion network to perform multi-source behavior response synchronization association processing on the head rotation orientation sequence, the gaze point spatial coordinate sequence, the button trigger intensity sequence, and the controller displacement trajectory sequence, generating a multi-source behavior response synchronization feature set, including: The head rotation orientation sequence, the gaze landing point spatial coordinate sequence, the button trigger intensity sequence, and the controller displacement trajectory sequence are segmented to generate a set of head rotation primitive fragments, a set of gaze dwell primitive fragments, a set of saccade jump primitive fragments, a set of force application gradual primitive fragments, a set of relaxation recovery primitive fragments, and a set of control trajectory primitive fragments. Based on the overlap of the time window, the head rotation primitive fragment set and the gaze persistence primitive fragment set are synchronously aligned to generate a head-eye synchronous coupling primitive pair set. Based on the overlap of the time window, the set of force gradual change primitive fragments and the set of control trajectory primitive fragments are synchronized and aligned to generate a set of manual control synchronous coupling primitive pairs. Cross-channel interaction and feature mapping processing are performed on the head-eye synchronization coupling primitive pair set and the hand-control synchronization coupling primitive pair set to generate a fused behavior representation vector. The fusion behavior representation vector is associated and encapsulated with the temporal boundary information of the head rotation primitive fragment set, the gaze persistence primitive fragment set, the force application gradient primitive fragment set, and the control trajectory primitive fragment set to generate the multi-source behavior response synchronization feature set. The method further includes: The set of head-eye synchronization coupling primitives is discretized by rotation amplitude and gaze region to generate a head-eye coordination state symbol sequence. The head-eye coordination state symbol sequence is segmented by a sliding time window to generate head-eye coordination behavior pattern units; Frequent pattern mining processing is performed on the head-eye coordination behavior pattern unit to obtain frequent head-eye coordination behavior pattern subsequences; A user head-eye coordination behavior habit model is generated based on the frequent head-eye coordination behavior pattern subsequences. The user head-eye coordination behavior habit model is applied to the predictive rendering process of virtual reality test scenarios, and rendering resources are preloaded for virtual objects in the predicted functional partitions of virtual reality test scenarios. Based on the frame rate data of the virtual reality test scene after preloading and processing rendering resources, a predictive rendering effect evaluation record is generated. The user head-eye coordination behavior habit model is bound and stored with the user identifier to construct a user head-eye coordination behavior habit model library.

2. The method according to claim 1, characterized in that, The step of mining user behavior feedback patterns based on the multi-source behavior response synchronization feature set to generate a behavior feedback response pattern description set includes: Extract the head-eye synchronization coupling primitive pair set from the multi-source behavior response synchronization feature set, perform gaze persistence duration statistical processing on each head-eye synchronization coupling primitive pair in the head-eye synchronization coupling primitive pair set, and calculate the time span of the gaze persistence primitive segment. Based on the comparison between the time span of the gaze persistence primitive segment and the preset gaze persistence duration threshold, the head-eye synchronization coupling primitive pair set is divided into a gaze persistence type coupling primitive pair subset and a gaze saccade type coupling primitive pair subset. For each gaze-pause type coupling primitive pair in the subset of gaze-pause type coupling primitive pairs, head micro-motion feature extraction processing is performed to extract the small angular offset fluctuation features of the head rotation orientation sequence during gaze-pause. The minute angular offset fluctuation characteristics are transformed from the time domain to the frequency domain to obtain the head micro-motion frequency domain energy distribution. Based on the energy distribution in the frequency domain of head micro-motion, the dominant frequency component of head micro-motion and the corresponding energy amplitude of the dominant frequency component of head micro-motion are extracted. The dominant frequency component of head micro-motion and the corresponding energy amplitude of the dominant frequency component of head micro-motion are used as the description of head micro-motion features in the gaze persistence state. For each gaze-pause type coupling primitive pair in the subset of gaze-pause type coupling primitive pairs, the gaze-pause point drift feature extraction process is performed to extract the gaze-pause point spatial coordinate drift trajectory of the aligned gaze-pause spatial coordinate sequence. The drift range of the spatial coordinate drift trajectory of the gaze point is quantified, the drift envelope area of ​​the spatial coordinate drift trajectory of the gaze point is calculated, and the drift envelope area is used as a description of the gaze stability feature under the gaze persistence state. The head micro-movement feature description and the gaze stability feature description are spliced ​​together to generate the gaze persistence mode description unit. Extract the set of manual synchronization coupling primitives from the set of multi-source behavioral response synchronization features, perform pressure rhythm analysis on each manual synchronization coupling primitive pair in the set of manual synchronization coupling primitives, and extract the duration of the rising edge of the pressure application gradient primitive segment and the duration of the falling edge of the pressure relaxation recovery primitive segment. A press control rhythm pattern vector is constructed based on the alternating sequence of the duration of the press rising edge and the duration of the press falling edge. The press control rhythm pattern vector is then subjected to periodic analysis to extract the dominant rhythm period of the press control rhythm pattern vector. The dominant rhythm period is then used as the description unit of the control rhythm pattern.

3. The method according to claim 1, characterized in that, The step of performing focus annotation processing on the raw behavioral data stream transmitted by the head-mounted display device according to the behavioral feedback response pattern description set to generate a test scenario behavioral feedback feature set with behavioral feedback pattern labels includes: A gaze focus indication vector is generated based on the gaze persistence mode description unit; The dominant rhythm cycle contained in the control rhythm pattern description unit is time-aligned with the gaze attention focus indicator vector to determine the overlapping time period of gaze persistence event and control rhythm event on the time axis. The head rotation orientation sequence segment and the gaze point spatial coordinate sequence segment are marked as high or low attention based on the overlapping time period. High-attention behavior feedback fragments are subjected to scene object reverse mapping processing to obtain a set of high-attention virtual object identifiers; Based on the cumulative gaze duration corresponding to each high-attention virtual object identifier in the set of high-attention virtual object identifiers, a virtual object attention priority sequence is generated; Based on the scene functional area category to which the preceding virtual object identifier in the virtual object attention priority sequence belongs, and the head micro-motion feature description in the gaze persistence mode description unit, a scene area attention behavior association mapping table is constructed. The high-attention behavior feedback segments, low-attention behavior feedback segments, virtual object attention priority sequence, and scene area attention behavior association mapping table are encapsulated and processed to generate the test scene behavior feedback feature set with behavior feedback pattern labels. The test scenario behavior feedback feature set with behavior feedback pattern labels is transmitted to the test analysis terminal.

4. The method according to claim 1, characterized in that, The method further includes: Extract the cross-channel behavior response co-occurrence matrix from the multi-source behavior response synchronization feature set, and normalize the co-occurrence intensity of the cross-channel behavior response co-occurrence matrix to obtain the normalized cross-channel behavior response co-occurrence matrix; The normalized cross-channel behavioral response co-occurrence matrix is ​​subjected to threshold filtering to extract head-eye-hand cross-channel ternary coupling patterns whose normalized co-occurrence intensity exceeds a preset co-occurrence intensity threshold. The head-eye-hand cross-channel ternary coupling pattern represents a composite behavioral response pattern in which head rotation behavior, gaze behavior, and hand manipulation behavior occur simultaneously. Obtain the head rotation primitive segment, gaze persistence primitive segment, and force gradient primitive segment corresponding to the head-eye-hand cross-channel ternary coupling mode. Take the intersection of the time intervals of the head rotation primitive segment, gaze persistence primitive segment, and force gradient primitive segment to obtain the ternary coupling time window. Within the three-element coupling time window, the control trajectory primitive fragments of the controller displacement trajectory sequence are extracted, and the control trajectory primitive fragments are classified into linear motion control trajectory primitives, arc motion control trajectory primitives, and polygonal motion control trajectory primitives. The kinematic parameters of the linear motion control trajectory primitive, the arc motion control trajectory primitive, and the polyline motion control trajectory primitive are calculated and processed respectively. The motion direction vector and motion speed sequence of the linear motion control trajectory primitive are extracted, the curvature radius sequence and angular velocity sequence of the arc motion control trajectory primitive are extracted, and the turning point coordinate sequence and turning angle sequence of the polyline motion control trajectory primitive are extracted. The gaze target virtual object is determined based on the gaze dwelling primitive fragment contained in the head-eye-hand cross-channel three-element coupling mode. The spatial position coordinates of the gaze target virtual object are compared with the motion direction vector of the linear motion control trajectory primitive to calculate the angular deviation between the gaze target direction and the control motion direction. The angular deviation, the motion rate sequence, the radius of curvature sequence, the angular velocity sequence, the turning point coordinate sequence, and the turning angle sequence are concatenated to generate a head-eye-hand ternary coupled control behavior feature vector. Cluster analysis is performed on the head-eye-hand ternary coupled manipulation behavior feature vectors corresponding to multiple ternary coupled time windows, and the head-eye-hand ternary coupled manipulation behavior feature vectors are aggregated into different manipulation behavior modal clusters. Each manipulation behavior modal cluster corresponds to a typical head-eye-hand coordinated manipulation behavior modality. Extract the cluster center feature vector of each control behavior mode cluster, and use the cluster center feature vector as the typical control behavior mode feature description of that control behavior mode; User control behavior modality fingerprints are constructed based on the proportion of occurrence of each typical control behavior modality feature description in the user group. The user control behavior modality fingerprints are then bound and stored with user identifiers to generate a user control behavior modality fingerprint database. The user control behavior modal fingerprint database is input into the adaptive adjustment module of the virtual reality test scene. The adaptive adjustment module performs personalized adaptation processing on the interaction mode and control sensitivity parameters in the virtual reality test scene according to the current user's user control behavior modal fingerprint.

5. The method according to claim 3, characterized in that, The method further includes: Obtain the gaze stability feature description corresponding to the high-attention behavior feedback segment; Based on the drift envelope area in the description of line-of-sight stability characteristics, a drift envelope area variation curve is generated; The drift envelope area change curve is subjected to trend decomposition processing to determine the period of visual fatigue accumulation; The distribution density of the low-attention behavior feedback segments during the visual fatigue accumulation period and the dominant rhythm cycle in the manipulation rhythm pattern description unit are obtained. Based on the proportion of the low-attention behavioral feedback segments to all behavioral feedback segments, and the offset ratio of the dominant rhythmic cycle relative to the baseline dominant rhythmic cycle, a comprehensive assessment vector for visual fatigue is constructed. The visual fatigue level assessment result of the current user is determined based on the comprehensive visual fatigue level assessment vector. Based on the visual fatigue level determination result, a visual stimulus intensity adjustment instruction for the virtual reality test scene is generated; The visual fatigue level determination result and the start and end timestamps of the visual fatigue accumulation period are encapsulated into a visual fatigue monitoring record, and the visual fatigue monitoring record is appended to the user behavior feedback feature database.

6. The method according to claim 1, characterized in that, The method further includes: Obtain the sequence fragment of the spatial coordinates of the gaze points in the set of head-eye synchronization coupling primitive pairs; Spatial trajectory smoothing and gaze shift recognition processing are performed on the spatial coordinate sequence segment of the gaze point to extract saccade jump primitive segments; Calculate the saccade jump vector based on the spatial coordinates of the starting and ending gaze points corresponding to the saccade jump primitive fragment. The virtual objects in the virtual reality test scene are constructed as a virtual object space topology graph, and the gazing jump vector is mapped to the virtual object space topology graph. Based on the mapping result of the saccadic jump vector on the virtual object space topology graph, construct the attention transfer probability matrix between virtual objects; The graph traversal analysis of the attention transfer probability matrix between the virtual objects is performed to obtain the dominant path of user attention transfer; The effectiveness of guidance in virtual reality test scenarios is evaluated by comparing the path similarity between the dominant user attention transfer path and the expected attention transfer path. Based on the deviated virtual object nodes in the dominant user attention transfer path that deviate from the expected attention transfer path, a scene guidance enhancement strategy is generated.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the force-gradient primitive fragment and the control trajectory primitive fragment from the set of manual control synchronous coupling primitive pairs; The pressing impact characteristics are extracted based on the force gradient primitive fragment, and the trajectory jitter characteristics are extracted based on the control trajectory primitive fragment; The pressing impact feature and the trajectory jitter feature are time-aligned and combined to generate a control stability feature tuple. Outlier detection processing is performed on the control stability feature sample set to identify outlier control stability feature pairs. Based on the time interval of the force-gradient primitive segment corresponding to the outlier control stability feature binary, the micro-period of control anomaly is determined. The sequence of head rotation orientation and the sequence of spatial coordinates of the gaze point within the micro-period of the abnormal manipulation are retrieved retrospectively, and head-eye stress coordination event recognition processing is performed. Based on the spatial coordinates of the termination gaze point in the head-eye stress coordination event, the interfering stimulus that caused the abnormal control was identified. The virtual object identifier of the interference stimulus, the time interval of the micro-period of the manipulation abnormality, and the binary tuple of the manipulation stability feature are encapsulated into a manipulation abnormality event record, and the manipulation abnormality event record is written into the interference source log database of the virtual reality test scene.

8. A user behavior feedback feature extraction system in a virtual reality testing scenario, characterized in that, Includes at least one service node; The service node includes a storage unit and a computing unit; the storage unit is used to store program code; the computing unit is used to run the program code to execute the user behavior feedback feature extraction method in the virtual reality test scenario according to any one of claims 1 to 7.

Images