Virtual reality human-computer interface interaction task quantitative evaluation system and method

Abstract

The present invention relates to a system and method for quantitatively evaluating interactive tasks in a virtual reality human-computer interface. The system comprises the following steps: interactive information collection, which collects information on operations in the action phase of a multimodal interactive process in a virtual environment to obtain an operation set; interactive information processing, which converts the operation set into a time-sequential action operator sequence, adds common operators to the sequence according to matching rules, assigns time values ​​to the common operators, and deletes redundant operators according to parallel time calculation rules to obtain an optimal operator sequence representing the optimal interactive process; interactive time calculation and estimation, which assigns time values ​​or time functions to all action operators in the optimal operator sequence to obtain an estimated time for the multimodal interactive process; interactive result output, which displays the optimal operator sequence and the corresponding interactive process; and efficiency evaluation based on the estimated time and displays the evaluation results. The present invention achieves a scientific evaluation of the efficiency of multimodal human-computer interaction.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction system evaluation, and in particular to a system and method for quantitatively evaluating virtual reality human-computer interface interaction tasks. Background Art

[0002] Virtual reality (VR) is a computer system that enables the creation and experience of virtual environments. It is computer-generated and interacts with users through sight, hearing, touch, and smell, creating an immersive, interactive visual simulation. Interactions in three-dimensional scenes are more complex, and the input and output methods are more diverse, involving multiple sensory channels such as vision, hearing, and touch. Multimodal interaction emphasizes the integration of human sensory channels, enabling people to acquire and transmit information through multiple channels. Touch interaction, natural speech, gestures, and eye tracking are all commonly used modalities that have demonstrated excellent results in actual interaction. Combining two or more input modes in a selective, sequential, concurrent, and complementary manner allows users to freely choose their preferred interaction method based on the situation, task, and context, enhancing the system's flexibility, naturalness, and efficiency. This increased flexibility is believed to improve the system's inclusiveness, efficiency, and freedom of movement.

[0003] Interactive tasks and interaction technologies in VR are tightly coupled, and the connection between tasks and technologies determines the efficiency of the interactive system. Interactive tasks refer to specific activities or operations that users need to complete, often requiring direct interaction with objects or systems in VR. Task complexity is also an important categorization dimension in VR. Simple tasks may consist of a single operation, while complex tasks may involve multi-step operations that integrate multiple modalities.

[0004] The development of interactive technology into three-dimensional virtual environments has also driven research into evaluation methods for human-computer interaction systems. Analytical evaluation, one of the primary methods for evaluating human-computer interaction processes, is based on task analysis and utilizes models. Compared to evaluating individual tasks separately, this method provides a more consistent evaluation system and facilitates quantitative characterization. Currently, most research focuses on using performance evaluation methods to assess individual tasks or operations in virtual reality, while some studies use analytical evaluation methods for evaluating individual tasks in virtual reality. This suggests a lack of universally applicable multimodal human-computer interface task evaluation schemes in virtual reality scenarios. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the present invention provides a virtual reality human-computer interface interaction task quantitative evaluation system and method, the purpose of which is to achieve virtual reality human-computer interface interaction efficiency evaluation.

[0006] The technical solution adopted in the present invention is as follows:

[0007] The present invention provides a method for quantitatively evaluating virtual reality human-computer interface interaction tasks, including interaction information collection, interaction information processing, interaction time estimation calculation, and interaction result output:

[0008] The interactive information collection includes:

[0009] Collect information about the operations in the action phase of the multimodal interaction process in the virtual environment; the operations in the action phase include a series of independent operations; the collected information includes the operation action, operation mode, mode, category and status information of the interactive object; the interactive object includes interface controls and physical controls;

[0010] Performing modal feature matching on the independent operation and the interactive object, and performing interaction determination, deleting operations that fail modal feature matching and / or interaction determination, and retaining valid operations;

[0011] For the valid operation, the start and end time of the operation is obtained according to the state information of the interactive object, and the series of independent operations are sorted in order of the start time to form an operation set that records the operation mode attributes, interactive object category attributes and actual operation time of the independent operation;

[0012] The interactive information processing includes:

[0013] Converting the independent operations in the operation set into corresponding action operators, thereby converting the operation set into an action operator sequence, wherein the action operator sequence includes an interaction operator and a pointing operator that may exist before the interaction operator; adding a general operator to the action operator sequence according to a first matching rule; assigning a preset time value to the general operator, and deleting redundant general operators according to a second matching rule, to obtain an optimal operator sequence representing an optimal interaction process;

[0014] The general operator corresponds to the operations in the remaining interaction phases except the action phase in the multimodal interaction process;

[0015] The interaction time estimation calculation includes:

[0016] Assigning a time value or a time function to all action operators in the optimal operator sequence to obtain an estimated time for this multimodal interaction process;

[0017] The interaction result output includes:

[0018] Displaying the optimal operator sequence and the corresponding interaction process through a display interface;

[0019] By comparing the actual operation time and the estimated time, the efficiency of the interaction process is evaluated, and the evaluation result is displayed.

[0020] Further technical solutions are:

[0021] Converting the independent operations in the operation set into corresponding action operators includes:

[0022] If the category and spatial position of the interaction object targeted by this independent operation are the same as those of the previous independent operation, convert this independent operation into an interaction operator;

[0023] If the spatial position or category of the interactive object targeted by this independent operation is different from that of the previous independent operation, convert this independent operation into a pointing operator plus an interactive operator;

[0024] determining a specific type of the interaction operator according to the type of the interaction object;

[0025] The specific type of the pointing operator is determined according to the operation mode of the independent operation.

[0026] The first matching rule includes:

[0027] If there is thinking before an action begins, add a general operator representing perception and cognition before the corresponding action operator;

[0028] If the operation mode is switched, add the general operators representing perception and cognition before the corresponding action operators;

[0029] If the interaction object is switched, add a general operator indicating perception before the corresponding pointing operator;

[0030] If an unexpected search occurs, add a general operator indicating perception before the corresponding pointing operator;

[0031] If the interactive operation ends, add a general operator indicating the system response after the last action operator;

[0032] The second matching rule includes:

[0033] Compare the duration of the universal operator representing the system response with the duration of the universal operator representing perception and cognition before the next operation, and delete the universal operator with shorter duration.

[0034] The obtaining the start and end time of the operation according to the state information of the interactive object includes:

[0035] Create a state script for each interactive object and define variables that represent changes in the interactive state.

[0036] Record the start and end time of each independent operation through the variable representing the change in the interaction state;

[0037] The variable representing the change in interaction state is a Boolean value.

[0038] The interactive information collection also includes: before interactively judging the independent operation, performing feature matching on the operation based on the operation action, operation mode, mode, category and status information of the interactive object, performing an initial evaluation of the validity of the operation, and deleting invalid operations.

[0039] The performing modal feature matching between the independent operation and the interactive object and performing interaction determination includes:

[0040] Perform modal feature matching on the independent operation and the interactive object: determine whether the modality of the interactive object corresponds to the operation modality of the independent operation and meets the set modality type, which includes near-field bare hand contact, far-field hand rays, far-field eye rays, gestures and voice.

[0041] Interaction determination is performed according to the following interaction determination rules:

[0042] The operation mode is near-field bare-hand contact. The judgment rule is: if the spatial position of the user's hand overlaps with the interface control or physical control, and the hand subsequently leaves the control's spatial position, it is considered an independent operation. If the interface control is a three-dimensional object and the physical control is a knob, a specified action must be performed.

[0043] The operation mode is far-field hand ray. The judgment rule is: in the pointing phase, the hand ray overlaps with the spatial position of the interface control. In the selection phase, a gesture to indicate confirmation is required, followed by cessation of the gesture. This is considered an independent operation.

[0044] The operation mode is far-field eye ray. The judgment rule is: in the pointing phase, the eye ray overlaps with the spatial position of the interface control. In the selection phase, an eye or gesture action is required to indicate confirmation, and then the action is stopped. This is considered an independent operation.

[0045] The operation mode is gesture action. The judgment rule is: the user makes a specific action or action sequence with his hand, then performs further operations in response to possible feedback from the system. After the execution is completed, the action or the last action of the action sequence is stopped and the user returns to the natural state. This is considered an independent operation.

[0046] The operation mode is voice, and the judgment rule is: after the voice input switch is turned on through the above four interaction modes or voice wake-up commands, the user starts voice input, waits for the set time after the input is completed, and then turns off the voice input switch, it is considered an independent operation.

[0047] The time function assigned to the action operator includes:

[0048] The time function T(P) representing the action operator P pointed by the interactive object in the near-field trigger area is: a+b*ID+c*cosθx+d*cosθy, where D is the distance between the user's hand and the interface control, W is the width of the interface control; θ is the deviation angle between the user's line of sight and the positive direction of the x-axis; where θ x is the horizontal deviation angle, θ y is the vertical deviation angle; a, b, c, d are coefficients respectively;

[0049] The first time function T(SH) = e*L+f and the second time function T(SV) = g*L+h of the continuous one-dimensional sliding action operator S of the interface control are represented. The two time functions respectively represent the time of movement in two mutually perpendicular directions, where L is the draggable length of the interface control; e, f, g, and h are coefficients.

[0050] The action operator M represents the free movement of the user's hand during move, rotate, and scale operations. H Time function T(M H )=Time_End–Time_Start, where Time_End and Time_Start are operator M respectively. H End time, start time;

[0051] Represents an action operator R that uses a ray to point to a 3D object within the trigger area P The time function T(R P )=i*ID+j*cosx+k*cosy+l, where D is the deviation angle between the interface controls, W is the width angle of the interface control, x is the horizontal deviation angle, y is the vertical deviation angle; i, j, k, l are coefficients;

[0052] The action operator R represents the process of the user moving towards the target direction of the finger ray T The time function T(R T ) = m*D / n, where D is the distance between the user's initial spatial position and the target control's spatial position in the virtual environment; m and n are coefficients;

[0053] Action operator T representing the process of the user turning the physical knob W The time function T(T W )=Time_End–Time_Start, where Time_End and Time_Start are operator T W End time, start time;

[0054] The time function T(T) of the action operator T that represents the user's natural gaze deviation towards the interaction target:

[0055] T(T)=s,D / W<7

[0056] T(T)=p*ID+q,D / W≥7

[0057] in, D is the deviation angle between the interface controls, W is the width of the interface control expressed in viewing angle; s, p, q are coefficients;

[0058] The action operator S represents the precise instruction of the user's voice input C Time function T(SC)=u*N S / v, where N S Refers to the number of syllables contained in a voice command; u and v are coefficients.

[0059] The displaying of the optimal operator sequence and the corresponding interaction process through a display interface includes:

[0060] Displaying each independent operation and the corresponding operator on the main interface in chronological order according to the optimal operator sequence;

[0061] Displaying information corresponding to the independent operation and the operator in the information area, including the start and end time and duration of the independent operation, the start and end time, duration and definition description of the operator;

[0062] Providing function buttons through the function area, and performing additional operations through the function buttons, including starting a new record, saving a record, and exporting a preset;

[0063] Starting a new record includes clearing all contents of the current main interface and starting a new record;

[0064] Said saving of records includes saving the optimal operator sequence and the interaction process corresponding thereto;

[0065] The export preset includes exporting information stored in the system, including operation information, operator types and time values.

[0066] The present invention also provides a virtual reality human-computer interface interaction task quantitative evaluation system, comprising:

[0067] The interactive information collection module is used to:

[0068] Collect information about the operations in the action phase of the multimodal interaction process in the virtual environment; the operations in the action phase include a series of independent operations; the collected information includes the operation action, operation mode, mode, category and status information of the interactive object; the interactive object includes interface controls and physical controls;

[0069] Performing modal feature matching on the independent operation and the interactive object, and performing interaction determination, deleting operations that fail modal feature matching and / or interaction determination, and retaining valid operations;

[0070] For the valid operation, the start and end time of the operation is obtained according to the state information of the interactive object, and the series of independent operations are sorted in order of the start time to form an operation set that records the operation mode attributes, interactive object category attributes and actual operation time of the independent operation;

[0071] Outputting the operation set to the interaction information processing module and the interaction result output module;

[0072] The interactive information processing module is used to:

[0073] Converting the independent operations in the operation set into corresponding action operators, thereby converting the operation set into an action operator sequence, and adding a general operator to the action operator sequence according to a first matching rule; assigning a preset time value to the general operator, and deleting redundant general operators according to a second matching rule to obtain an optimal operator sequence representing an optimal interaction process; the general operator corresponds to the operations in the remaining interaction phases of the current multimodal interaction process except the action phase;

[0074] outputting the optimal operator sequence to an interaction time estimation module;

[0075] The interaction time estimation module assigns a time value or a time function to all action operators in the optimal operator sequence to obtain an estimated time for this multimodal interaction process;

[0076] The interaction result output module is used to:

[0077] Displaying the optimal operator sequence and the corresponding interaction process through a display interface;

[0078] By comparing the actual operation time and the estimated time, the efficiency of the interaction process is evaluated, and the evaluation result is displayed.

[0079] The beneficial effects of the present invention are as follows:

[0080] This invention provides a quantitative assessment solution for common interactive tasks in virtual reality, such as navigation, selection, manipulation, and system control. By estimating the time of the interaction process, it achieves efficient assessment of interaction efficiency and ensures consistency in the assessment. Specifically, it has the following advantages:

[0081] (1) The present invention disassembles the multimodal interaction of the existing virtual environment interface, and realizes real-time time estimation of the interaction process through operation set collection, operator sequence extraction, operator sequence addition and deletion, and time setting. The interaction efficiency is evaluated based on the time estimation, and the visual expression of the perception, cognition, and motor behavior disassembly of the interaction process is realized.

[0082] (2) In the information collection stage, the present invention only collects the action process and ignores perception, cognition and system response. In the processing stage, the independent operations are converted into corresponding action operators by splitting the operation set, and then general operators that reflect perception, cognition and system response are added to the action operator sequence, thereby supplementing all interaction stages and greatly reducing the difficulty and cost of information collection.

[0083] (3) During the information collection phase, the present invention determines the validity of operations through modal feature matching, deletes invalid operations, and ensures that all operations in the subsequently output operation set are valid.

[0084] (4) The present invention deletes redundant operators according to the time value of the general operator using the corresponding matching rules, obtains the optimal operator sequence representing the optimal interaction process, improves the time estimation accuracy of the optimal interaction process, and provides a reliable evaluation basis for interaction efficiency evaluation.

[0085] (5) The present invention displays the real-time interactive operation process and efficiency evaluation results, making it easy to observe the interactive situation intuitively and to modify and optimize the evaluation results by changing the preset information.

[0086] Other features and advantages of the present invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 Schematic diagram of the process of the embodiment of the present invention.

[0088] Figure 2 This is a schematic diagram of the hierarchical division structure of various types of multimodal interactions in the virtual environment proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0089] The specific embodiments of the present invention are described below with reference to the accompanying drawings.

[0090] See also Figure 1 A quantitative evaluation method for a virtual reality human-computer interface interaction task in this embodiment includes the following steps:

[0091] S1. Interaction information collection, including:

[0092] S11. Collecting information about operations in the action phase of the multimodal interaction process in the virtual environment; the operations in the action phase include a series of independent operations; the collected information includes operation actions, operation modes, and the modes, categories, and status information of interactive objects; the interactive objects include interface controls and physical controls;

[0093] S12. Performing modal feature matching on the independent operation and the interactive object, and performing interaction determination, deleting operations that fail modal feature matching and / or interaction determination, and retaining valid operations;

[0094] S13. For the valid operation, obtain the start and end time of the operation according to the state information of the interactive object, sort the series of independent operations in order of the start time, and form an operation set that records the operation mode attributes, interactive object category attributes and actual operation time of the independent operation.

[0095] The interaction phase (i.e., the human-computer interaction phase) in the multimodal interaction process includes perception, cognition, action, and system response. Perception and cognition mainly reflect the user's perception and cognitive process during the interaction; action reflects the user's action behavior, which can be further divided into three categories: hand movements, eye movements, and voice; and system response reflects the machine's response process. In step S11, only the operation information of the action phase is collected to form the final operation set. This can avoid the problem of difficulty and high cost in capturing the perception and cognitive processes through external devices.

[0096] See also Figure 2 , various types of multimodal interactions in the virtual environment can be divided into three levels, namely the task layer, the operation layer and the operator layer. The tasks in the task layer refer to the interactive tasks that need to be completed, such as editing and deleting documents. It consists of a series of operations. The figure shows the four common types of interactive tasks in virtual reality. The operations in the operation layer, such as opening a page, include sequence action behaviors. When this embodiment collects the operation information of the action stage of the interaction process, the sequence action behaviors are divided into independent operations. The operators contained in the operator layer represent indivisible action behaviors, such as pinching and clicking, which can be standardized and assigned.

[0097] Among them, the modal feature reflects the modal type of the operation mode. In this embodiment, the operation modal types include near-field bare hand contact, far-field hand rays, far-field eye rays, gestures and voice. In order to collect effective independent operations, on the one hand, it is necessary to identify whether the modality of the interactive object matches the modality of the independent operation. For example, the interactive object only supports near-field bare hand contact, and the independent operation for interacting with it is far-field hand rays, then the two do not match, and the operation is regarded as an invalid operation. On the other hand, it is necessary to identify whether the interactive operation is standard, that is, whether the operation meets the corresponding interaction judgment rules. In step S12, the independent operation is matched with the interactive object for modal features, and interaction judgment is performed to identify the operation to retain valid operations and delete invalid operations, as follows:

[0098] On the one hand, the modal feature matching of the independent operation and the interactive object is performed: whether the modality of the interactive object corresponds to the operation modality of the independent operation and meets the set modality type, if so, the match is consistent and the operation is deemed valid, otherwise it is deemed invalid

[0099] On the other hand, interaction determination is performed according to the following interaction determination rules:

[0100] The operation mode is near-field bare-hand contact, and the judgment rule is: if the spatial position of the user's hand overlaps with the interface control or physical control, and the hand subsequently leaves the control's spatial position, it is considered an independent operation; if the interface control is a three-dimensional object, a specified action is required, such as pinching the thumb and index finger; if the physical control is a knob, a specified action is required, such as pinching and rotating;

[0101] The operation mode is far-field hand ray. The judgment rule is: in the pointing phase, the hand ray overlaps with the spatial position of the interface control. In the selection phase, a gesture to indicate confirmation is required, such as pinching the thumb and index finger, and then stopping the gesture. This is considered an independent operation.

[0102] The operation mode is far-field eye ray, and the judgment rules are as follows: in the pointing phase, the eye ray overlaps with the spatial position of the interface control. In the selection phase, eye or gesture movements are required to indicate confirmation. For example, eye movements include blinking, gaze duration reaching a specific value, etc. Gesture movements such as pinching the thumb and index finger and then stopping the movement are considered as independent operations.

[0103] The operation mode is a gesture action. The judgment rule is: the user makes a specific movement or action sequence with their hand, and then takes further action in response to the feedback provided by the system, such as performing an action on a pop-up menu. After the action is completed, the action is stopped or the last action of the action sequence returns to the natural state. This is considered an independent operation.

[0104] The operation mode is voice, and the judgment rule is: after the voice input switch is turned on through the above four interaction modes or voice wake-up commands, the user starts voice input, waits for the set time after the input is completed, and then turns off the voice input switch, it is considered an independent operation.

[0105] The highest goal of this embodiment is efficiency evaluation, so it is necessary to record the actual operation time. In step S13, the start and end time of the operation is obtained according to the state information of the interactive object, including:

[0106] Create a state script for each interactive object and define variables that represent changes in the interactive state.

[0107] The start and end time of each independent operation is recorded through the variable representing the change in the interaction state.

[0108] In a specific embodiment, the variable representing the interaction state change is a Boolean value with two states: true and false. True indicates that the interaction object is selected, and false indicates that the interaction object is not selected. By recording the state change time of the interaction object, the start and end time of the corresponding interaction operation can be obtained.

[0109] Through the information collection, modal feature matching, and time recording in step S1 of this embodiment, a set of all operations in the action phase of the current interaction process is ultimately obtained, sorted by start time. Each operation in the set includes its own operation modality attributes or interaction object category attributes, as well as its execution start and end times. The output operation set provides the modality attributes, interaction object category information, and time sequence information required for subsequent operation splitting by the interaction processing module.

[0110] As an improvement to the above embodiment, step S1 of this embodiment, before executing step S12, further includes:

[0111] The operation is feature matched based on the operation action, operation mode, mode, category and state information of the interactive object, the validity of the operation is initially evaluated, and invalid operations are deleted.

[0112] As a specific implementation method, the collected operation actions, operation modes, modes, categories and status information of interactive objects are matched one by one through preset information. If they are all matched, it means that the operation is valid, otherwise it is invalid. It can be understood that the preset information contains the operation actions, operation modes, modes, categories and status of interactive objects, and sets corresponding features. For example, the operation actions included in the preset information include pinching and clicking, and the collected operation action is slapping, which is not within the preset action range, then the operation is considered invalid. The preset information can be specifically decomposed by a large number of interactive task sets, such as Figure 1The "covered interactive task set" shown in is obtained, and the various attributes and categories of tasks, operations and operators in the preset information of this method are set with reference to the covered interactive task set.

[0113] S2. Interactive information processing, including:

[0114] The independent operations in the operation set are converted into corresponding action operators, thereby converting the operation set into an action operator sequence, and adding a general operator to the action operator sequence according to a first matching rule; assigning a preset time value to the general operator, and deleting redundant general operators according to a second matching rule to obtain an optimal operator sequence representing an optimal interaction process; the general operator corresponds to the operations in the remaining interaction phases of the current multimodal interaction process except the action phase.

[0115] Step S2 splits the information in the operation set outputted in step S1 to obtain action operators arranged in time sequence, namely the operator sequence, which includes the interaction operator and the pointing operator that may exist before the interaction operator.

[0116] Converting the independent operations in the operation set into corresponding action operators includes:

[0117] If the category and spatial position of the interaction object targeted by this independent operation are the same as those of the previous independent operation, convert this independent operation into an interaction operator;

[0118] If the spatial position or category of the interactive object targeted by this independent operation is different from that of the previous independent operation, convert this independent operation into a pointing operator plus an interactive operator;

[0119] determining a specific type of the interaction operator according to the type of the interaction object;

[0120] The specific type of the pointing operator is determined according to the operation mode of the independent operation.

[0121] If the category of the interaction object targeted by the current independent operation is the same as that of the previous independent operation, the current independent operation corresponds to the interaction operator; otherwise, it corresponds to the pointing operator.

[0122] Among them, interactive operators include, for example, continuous one-dimensional sliding of interactive interface controls, clicking of interactive interface controls, etc.; pointing operators include, for example, pointing of interactive objects in the near-field trigger area, pointing of interactive objects in the far-field trigger area using rays, etc.

[0123] Specifically, the interactive object categories of this embodiment include categories of interactive interface controls, mainly including buttons, sliders, windows, three-dimensional objects, etc., where three-dimensional objects can be moved, rotated or scaled; and also include categories of physical controls, including knobs, buttons, etc.

[0124] The specific rules for determining the two types of action operators in the action operator sequence include:

[0125] a) If the interactive object of this independent operation is a window or button, it is considered a normal click, and the interaction operator "C" or "F" is used according to the difference between near-field and far-field interactions. C ”;

[0126] b) If the interactive object of this independent operation is a slider, use the interactive operator "S";

[0127] c) If the interactive object of this independent operation is a three-dimensional object, use the interactive operator "M H ”;

[0128] d) If this independent operation is a near-field bare-hand contact mode, add a pointing operator "P" before the corresponding interaction operator;

[0129] e) If this independent operation is the far-field hand ray mode, add the pointing operator "R" before the corresponding interaction operator. P ”;

[0130] f) If this independent operation is a far-field eye ray mode, use the pointing operator "T".

[0131] As mentioned above, this embodiment only collects operations in the action stage during the collection phase in step S1, but does not collect perception, cognition, and system response. Therefore, in this step, the first matching rule is used to insert universal operators corresponding to the perception, cognition, and system response into the action operator sequence, i.e., before and after one or some action operators, to supplement the perception and cognition process.

[0132] Among them, the universal operators include perception operators, cognitive operators and system response operators corresponding to perception, cognition and system response respectively.

[0133] The first matching rule includes:

[0134] If there is thinking before an action begins, add the general operator "AE" representing perception and cognition before the corresponding action operator;

[0135] If the operation mode is switched, add the general operator "AE" representing perception and cognition before the corresponding action operator;

[0136] If the interaction object is switched, add the universal operator "E" representing perception before the corresponding pointing operator;

[0137] If an unexpected search occurs, such as an object not appearing in the given spatial location, add the general operator "E" to indicate perception before the corresponding pointing operator. S ”;

[0138] If the interactive operation ends, add the general operators "R1" and "R2" indicating the system response after the last action operator.

[0139] The description of various operators set in this embodiment is shown in Table 1 below.

[0140] Table 1 Operator Description

[0141]

[0142]

[0143] The system response operators include the computer's run operator "R1" and the feedback operator "R2." This embodiment considers the possible overlap between computer operation and feedback and user interaction, and implements parallel time calculation rules. Therefore, the pre-stored time values ​​of general operators (i.e., the duration required to execute the corresponding operation) are read, and redundant general operators are deleted according to the second matching rule to obtain the optimal operator sequence representing the optimal interaction process.

[0144] The second matching rule includes:

[0145] Compare the general operators that represent the system response (including "R1", "R2" as shown in Table 1) with the general operators that represent the perception and cognition before the next operation (including "E", "E S ”, “A”) and delete the general operators with shorter duration.

[0146] Step S2 is to split the operation set of step S1 into corresponding operators, and process them to form a time-optimized optimal operator sequence that can represent the optimal interaction process.

[0147] The optimal operator sequence and the number of operators it contains are both manifestations of interaction rules, and the interaction rules can realize the reverse process from operator sequence to operation and then to task.

[0148] S3. Interaction time estimation calculation, including:

[0149] All action operators in the optimal operator sequence are assigned a time value or a time function to obtain a computational estimated time for this multimodal interaction process.

[0150] As shown in Table 1, the time functions of each action operator include:

[0151] The time function T(P) representing the action operator P pointed by the interactive object in the near-field trigger area is: a+b*ID+c*cosθx+d*cosθy, where D is the distance between the user's hand and the spatial position of the interface control, W is the width of the interface control, and θ is the angle between the user's line of sight and the positive direction of the x-axis, where θ x is the horizontal deviation angle, θ y is the vertical deviation angle; a, b, c are the coefficients in the formula. Through experimental measurement, the reference values ​​of a, b, c are: 1648, 121, -177, -780 respectively;

[0152] The first time function T(SH) = a*L+b and the second time function T(SV) = c*L+d representing the continuous one-dimensional sliding action operator S of the interface control respectively represent the time of movement in two mutually perpendicular directions, where L is the draggable length of the interface control; a, b, and c are the coefficients in the formula;

[0153] The reference values ​​of a, b, c, and d in this embodiment are 96, 560, 86, and 444, respectively.

[0154] The action operator M represents the free movement of the user's hand during move, rotate, and scale operations. H Time function T(M H )=Time_End–Time_Start, where Time_End and Time_Start are operator M respectively. H End time, start time;

[0155] Represents an action operator R that uses a ray to point to a 3D object within the trigger area P The time function T(R P )=a*ID+b*cosx+c*cosy+d, where D is the deviation angle between the interface controls, W is the width angle of the interface control, x is the horizontal deviation angle, and y is the vertical deviation angle; a, b, and c are the coefficients in the formula;

[0156] The experimental determination of this embodiment yielded reference values ​​of a, b, c, and d as follows: 243, 676, 973, and -1679, respectively;

[0157] The action operator R represents the process of the user moving towards the target direction of the finger ray T The time function T(R T) = a*D / b, where D is the distance between the user's initial spatial position and the target control's spatial position in the virtual environment; a and b are coefficients in the formula;

[0158] The experimental results of this embodiment show that the reference values ​​of a and b are 1000 and 2.81 respectively.

[0159] Action operator T representing the process of the user turning the physical knob W The time function T(T W )=Time_End–Time_Start, where Time_End and Time_Start are operator T W End time, start time;

[0160] The time function T(T) of the action operator T that represents the user's natural gaze deviation towards the interaction target:

[0161] T(T)=a,D / W<7

[0162] T(T)=b*ID+c,D / W≥7

[0163] in, D is the deviation angle between the interface controls, W is the width of the interface control expressed in viewing angle; a, b, c are the coefficients in the formula;

[0164] The experimental results of this embodiment show that the reference values ​​of a, b, and c are 300, 949, and -2481, respectively.

[0165] The action operator S represents the precise instruction of the user's voice input C Time function T(SC)=a*N S / b, where N S Refers to the number of syllables contained in a voice command; a and b are coefficients in the formula. Through experimental measurement, the reference values ​​of a and b are 1000 and 3.57 respectively.

[0166] It can be seen that the above operators P and R P The time function value of T is mainly calculated by the spatial position and size of the interface control. The time function value of operator S is calculated by the front and back position and direction of the slider. h The time function value of is calculated from the start and end time of the interaction.

[0167] The duration of other action operators is assigned according to the fixed time values ​​set in Table 1.

[0168] The optimal operator sequence assigned with time values ​​obtained in this step realizes the time estimation of each operation and provides a basis for subsequent interaction evaluation.

[0169] S4. Interaction result output, including:

[0170] On the one hand, the optimal operator sequence and the corresponding interaction process are displayed through a display interface, which includes:

[0171] According to the optimal operator sequence, corresponding independent operations and operators of different categories are displayed on the main interface in chronological order.

[0172] Displaying information corresponding to the independent operation and the operator in the information area, including the start and end time and duration of the corresponding independent operation, and the start and end time, duration and definition description of the corresponding operator;

[0173] The function area provides function buttons, through which the user can also perform additional operations, including starting a new record, saving a record, and exporting a preset;

[0174] New record means clearing all contents of the current main interface and starting a new record; Save record means saving the optimal operator sequence and its corresponding interaction process in the form of a table; Export preset means exporting the information stored in the system in the form of a table, including operation information, operator type and its time value.

[0175] On the other hand, by comparing the actual operation time and the estimated time, the efficiency of the interaction process is evaluated, and the evaluation result is displayed.

[0176] Specifically, based on the set of operations obtained in step S1, the actual execution time of each individual operation is obtained. The efficiency of the interaction process is evaluated based on the difference between the actual execution time and the estimated time obtained in step S3. Since operation times are measured in seconds and are significantly affected by noise, if the difference is within 10% for a specific operation, the operation execution is considered reasonable and the efficiency evaluation result is good. Otherwise, the evaluation result is poor.

[0177] As an improved implementation manner, the interaction result output further includes:

[0178] The operation record area presents the current operation information and records the user's functional operations in real time, indicating the current system operation status.

[0179] See also Figure 1 This embodiment also provides a virtual reality human-computer interface interaction task quantitative evaluation system, including:

[0180] The interactive information collection module is used to:

[0181] Collect information about the operations in the action phase of the multimodal interaction process in the virtual environment; the operations in the action phase include a series of independent operations; the collected information includes the operation action, operation mode, mode, category and status information of the interactive object; the interactive object includes interface controls and physical controls;

[0182] Performing modal feature matching on the independent operation and the interactive object, and performing interaction determination, deleting operations that fail modal feature matching and / or interaction determination, and retaining valid operations;

[0183] For the valid operation, the start and end time of the operation is obtained according to the state information of the interactive object, and the series of independent operations are sorted in order of the start time to form an operation set that records the operation mode attributes, interactive object category attributes and actual operation time of the independent operation;

[0184] Outputting the operation set to the interaction information processing module and the interaction result output module;

[0185] The interactive information processing module is used to:

[0186] Converting the independent operations in the operation set into corresponding action operators, thereby converting the operation set into an action operator sequence, and adding a general operator to the action operator sequence according to a first matching rule; assigning a preset time value to the general operator, and deleting redundant general operators according to a second matching rule to obtain an optimal operator sequence representing an optimal interaction process; the general operator corresponds to the operations in the remaining interaction phases of the current multimodal interaction process except the action phase;

[0187] Outputting the optimal operator sequence to an interaction time estimation calculation module;

[0188] The interaction time estimation calculation module assigns a time value or a time function to all action operators in the optimal operator sequence to obtain an estimated time for this multimodal interaction process;

[0189] The interaction result output module is used to:

[0190] Displaying the optimal operator sequence and the corresponding interaction process through a display interface;

[0191] By comparing the actual operation time and the estimated time, the efficiency of the interaction process is evaluated, and the evaluation result is displayed.

[0192] Those skilled in the art will understand that the foregoing descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will be able to modify the technical solutions described in the foregoing embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A quantitative evaluation method for virtual reality human-computer interface interaction tasks, characterized by: Including interaction information collection, interaction information processing, interaction time estimation calculation and interaction result output: The interactive information collection includes: Collect information about the actions in the action phase of the multimodal interaction process in the virtual environment; the actions in the action phase include a series of independent operations; the collected information includes the operation action, the operation mode, the mode, category, and status information of the interactive object; the interactive object includes interface controls and physical controls; Performing modal feature matching on the independent operation and the interactive object, and performing interaction determination, deleting operations that fail modal feature matching and / or interaction determination, and retaining valid operations; For the valid operation, the start and end time of the operation is obtained according to the state information of the interactive object, and the series of independent operations are sorted in order of the start time to form an operation set that records the operation mode attributes, interactive object category attributes and actual operation time of the independent operation; The interactive information processing includes: Converting the independent operations in the operation set into corresponding action operators, thereby converting the operation set into an action operator sequence, wherein the action operator sequence includes an interaction operator and a pointing operator that may exist before the interaction operator; adding a general operator to the action operator sequence according to a first matching rule; assigning a preset time value to the general operator, and deleting redundant general operators according to a second matching rule, to obtain an optimal operator sequence representing an optimal interaction process; The general operator corresponds to the operations in the remaining interaction phases except the action phase in the multimodal interaction process; The interaction time estimation calculation includes: Assigning a time value or a time function to all action operators in the optimal operator sequence to obtain an estimated time for this multimodal interaction process; The interaction result output includes: Displaying the optimal operator sequence and the corresponding interaction process through a display interface; By comparing the actual operation time and the estimated time, the efficiency of the interaction process is evaluated, and the evaluation result is displayed.

2. The method according to claim 1, characterized in that Converting the independent operations in the operation set into corresponding action operators includes: If the category and spatial position of the interaction object targeted by this independent operation are the same as those of the previous independent operation, convert this independent operation into an interaction operator; If the spatial position or category of the interactive object targeted by this independent operation is different from that of the previous independent operation, convert this independent operation into a pointing operator plus an interactive operator; determining a specific type of the interaction operator according to the type of the interaction object; The specific type of the pointing operator is determined according to the operation mode of the independent operation.

3. The method according to claim 2, characterized in that The first matching rule includes: If there is thinking before an action begins, add a general operator representing perception and cognition before the corresponding action operator; If the operation mode is switched, add the general operators representing perception and cognition before the corresponding action operators; If the interaction object is switched, add a general operator indicating perception before the corresponding pointing operator; If an unexpected search occurs, add a general operator indicating perception before the corresponding pointing operator; If the interactive operation ends, add a general operator indicating the system response after the last action operator; The second matching rule includes: Compare the duration of the universal operator representing the system response with the duration of the universal operator representing perception and cognition before the next operation, and delete the universal operator with shorter duration.

4. The method according to claim 1, wherein The obtaining the start and end time of the operation according to the state information of the interactive object includes: Create a state script for each interactive object and define variables that represent changes in the interactive state. Record the start and end time of each independent operation through the variable representing the change in the interaction state; The variable representing the change in interaction state is a Boolean value.

5. The method according to claim 1, wherein The interactive information collection also includes: before interactively judging the independent operation, performing feature matching on the operation based on the operation action, operation mode, mode, category and status information of the interactive object, performing an initial evaluation of the validity of the operation, and deleting invalid operations.

6. The method according to claim 1, characterized in that The performing modal feature matching between the independent operation and the interactive object and performing interaction determination includes: Perform modal feature matching on the independent operation and the interactive object: determine whether the modality of the interactive object corresponds to the operation modality of the independent operation and meets the set modality type, which includes near-field bare hand contact, far-field hand rays, far-field eye rays, gestures and voice.

7. The method according to claim 6, characterized in that Interaction determination is performed according to the following interaction determination rules: The operation mode is near-field bare-hand contact. The judgment rule is: if the spatial position of the user's hand overlaps with the interface control or physical control, and the hand subsequently leaves the control's spatial position, it is considered an independent operation. If the interface control is a three-dimensional object and the physical control is a knob, a specified action must be performed. The operation mode is far-field hand ray. The judgment rule is: in the pointing phase, the hand ray overlaps with the spatial position of the interface control. In the selection phase, a gesture to indicate confirmation is required, followed by cessation of the gesture. This is considered an independent operation. The operation mode is far-field eye ray. The judgment rule is: in the pointing phase, the eye ray overlaps with the spatial position of the interface control. In the selection phase, an eye or gesture action is required to indicate confirmation, and then the action is stopped. This is considered an independent operation. The operation mode is gesture action. The judgment rule is: the user makes a specific action or action sequence with his hand, then performs further operations in response to possible feedback from the system. After the execution is completed, the action or the last action of the action sequence is stopped and the user returns to the natural state. This is considered an independent operation. The operation mode is voice, and the judgment rule is: after the voice input switch is turned on through the above four interaction modes or voice wake-up commands, the user starts voice input, waits for the set time after the input is completed, and then turns off the voice input switch, it is considered an independent operation.

8. The method according to claim 1, characterized in that The time function assigned to the action operator includes: The time function of the action operator P pointed by the interactive object in the near-field trigger area is T(P) = a+b*ID+c*cosθx+d*cosθy, where θ is the width of the object; θ is the deviation angle between the user's line of sight and the positive direction of the x-axis; where θ x is the horizontal deviation angle, θ y is the vertical deviation angle; a, D is the distance between the user’s hand and the interface control, W is the coefficient of the interface control b, c, and d respectively; The first time function T(SH) = e*L+f and the second time function T(SV) = g*L+h of the continuous one-dimensional sliding action operator S of the interface control are represented. The two time functions respectively represent the time of movement in two mutually perpendicular directions, where L is the draggable length of the interface control; e, f, g, and h are coefficients. The action operator M represents the free movement of the user's hand during move, rotate, and scale operations. H Time function T(M H )=Time_End–Time_Start, where Time_End and Time_Start are operator M respectively. H End time, start time; Represents an action operator R that uses a ray to point to a 3D object within the trigger area P The time function T(R P )=i*ID+j*cosx+k*cosy+l, where D is the deviation angle between the interface controls, W is the width angle of the interface control, x is the horizontal deviation angle, y is the vertical deviation angle; i, j, k, l are coefficients; The action operator R represents the process of the user moving towards the target direction of the finger ray T The time function T(R T ) = m*D / n, where D is the distance between the user's initial spatial position and the target control's spatial position in the virtual environment; m and n are coefficients; Action operator T representing the process of the user turning the physical knob W The time function T(T W )=Time_End–Time_Start, where Time_End and Time_Start are operator T W End time, start time; The time function T(T) of the action operator T that represents the user's natural gaze deviation towards the interaction target: T(T)=s,D / W<7 T(T)=p*ID+q,D / W≥7 in, D is the deviation angle between the interface controls, W is the width of the interface control expressed in viewing angle; s, p, q are coefficients; The action operator S represents the precise instruction of the user's voice input C Time function T(SC)=u*N S / v, where N S Refers to the number of syllables contained in a voice command; u and v are coefficients.

9. The method according to claim 1, characterized in that The displaying of the optimal operator sequence and the corresponding interaction process through a display interface includes: Displaying each independent operation and the corresponding operator on the main interface in chronological order according to the optimal operator sequence; Displaying information corresponding to the independent operation and the operator in the information area, including the start and end time and duration of the independent operation, the start and end time, duration and definition description of the operator; Providing function buttons through the function area, and performing additional operations through the function buttons, including starting a new record, saving a record, and exporting a preset; Starting a new record includes clearing all contents of the current main interface and starting a new record; Said saving of records includes saving the optimal operator sequence and the interaction process corresponding thereto; The export preset includes exporting information stored in the system, including operation information, operator types and time values.

10. A virtual reality human-computer interface interaction task quantitative evaluation system, characterized in that: include: The interactive information collection module is used to: Collect information about the actions in the action phase of the multimodal interaction process in the virtual environment; the actions in the action phase include a series of independent operations; the collected information includes the operation action, the operation mode, the mode, category, and status information of the interactive object; the interactive object includes interface controls and physical controls; Performing modal feature matching on the independent operation and the interactive object, and performing interaction determination, deleting operations that fail modal feature matching and / or interaction determination, and retaining valid operations; For the valid operation, the start and end time of the operation is obtained according to the state information of the interactive object, and the series of independent operations are sorted in order of the start time to form an operation set that records the operation mode attributes, interactive object category attributes and actual operation time of the independent operation; Outputting the operation set to the interaction information processing module and the interaction result output module; The interactive information processing module is used to: Converting the independent operations in the operation set into corresponding action operators, thereby converting the operation set into an action operator sequence, and adding a general operator to the action operator sequence according to a first matching rule; assigning a preset time value to the general operator, and deleting redundant general operators according to a second matching rule to obtain an optimal operator sequence representing an optimal interaction process; the general operator corresponds to the operations in the remaining interaction phases of the current multimodal interaction process except the action phase; outputting the optimal operator sequence to an interaction time estimation module; The interaction time estimation module assigns a time value or a time function to all action operators in the optimal operator sequence to obtain an estimated time for this multimodal interaction process; The interaction result output module is used to: Displaying the optimal operator sequence and the corresponding interaction process through a display interface; By comparing the actual operation time and the estimated time, the efficiency of the interaction process is evaluated, and the evaluation result is displayed.

Images