Interaction system virtual test method and device based on multi-modal cognitive twin

Abstract

The invention provides an interactive system virtual test method and device based on multi-mode cognitive twin bodies. The method comprises the following steps: 1) constructing user cognitive twin: outputting a behavior decision tree with confidence through an adversarial generative network based on population attributes, cognitive preferences and behavior history; 2) creating a dynamic interaction environment: encoding a to-be-tested interface into a semantic graph structure which can be analyzed by a machine, and associating a physical operation constraint rule base; 3) running a multi-agent test: deploying a twin cluster in the virtual sandbox, and recording an operation path, an attention thermodynamic diagram and decision delay; and 4) generating a design defect map, namely, positioning an interface cognitive friction point by comparing the deviation degree of an expected interaction flow and an actual behavior track. According to the method, the problems that a traditional user test period is long and scene coverage is limited are solved, and early interception of interaction risks and design optimization preposition are achieved. The method can be widely applied to high-risk / high-cost / high-cycle test scenes of the man-machine interaction system.

Description

Technical Field

[0001] This invention lies at the intersection of human-computer interaction (HCI) testing and artificial intelligence (AI) simulation technology, specifically a method and device for testing interactive systems based on virtual digital humans. By constructing a multimodal cognitive twin to simulate real user behavior, it enables automated, highly concurrent testing of interactive interfaces, addressing the high cost, large sample bias, and limited scenario coverage of traditional user testing. Background Art

[0002] The limitations of existing technologies related to using digital humans to create virtual samples are: 1) Digital human modeling is limited. While the cited patent CN119864164A (Intelligent Health Digital Human Model) can generate health intervention plans based on physiological and psychological data, its model lacks dynamic simulation of the user's cognitive decision-making process, making it unable to adapt to the modeling requirements of complex factors such as behavioral preferences and cognitive misjudgments in interactive interface testing. Similarly, patent CN118535005A (Virtual Digital Human Interactive Device) relies on a preset voice / visual wake-up process, limiting interactive responses to fixed scripts and failing to autonomously generate diverse decision paths. 2) Static test environment. Virtual sample generation technology in the medical imaging field (e.g., CN116563246A) only supports static data acquisition (e.g., CT image projection) and does not establish a dynamic coupling mechanism between the interactive interface and user behavior, resulting in test results that are out of touch with real-world operating scenarios. 3) One-sidedness of evaluation metrics. Existing digital human systems (such as CN119902625A) focus on emotion recognition (voice / expression analysis) but ignore cognitive friction (such as operation path regression and attention distraction), a core interactive experience metric.

[0003] Industry pain points include: 1) Traditional A / B testing requires recruiting real users, takes up to 3-4 weeks, and covers limited scenarios (e.g., special populations and extreme environments are difficult to simulate); 2) The interaction needs of groups such as those with severe disabilities are difficult to verify through physical testing (for example, a breathing-controlled HMI system requires customized equipment).

[0004] The differences between this invention and existing patents are shown in Table 1: Table 1: Comparison of differences between this invention and existing patents Dimensions CN119864164A (Healthy Digital Human) CN118535005A (Interactive Digital Human) The present invention Modeling Core Physiological indicator intervention plan Fixed script response Cognitive-Behavioral Decision Chain Environmental Interaction none Single device wakeup Dynamic coupling of multimodal interface semantics Evaluation Metrics Health status prediction accuracy Interaction Fluency Cognitive friction coefficient Application Scenario Health Management Customer Service Dialogue Interactive system risk pre-verification Summary of the Invention

[0005] This invention provides a method and apparatus for virtual testing of interactive systems based on multimodal cognitive twins. The method includes: constructing a user cognitive twin, using a generative adversarial network to output a confidence-based behavioral decision tree based on demographic attributes, cognitive preferences, and behavioral history; creating a dynamic interactive environment, encoding the interface to be tested into a machine-parseable semantic graph structure, and associating it with a physical operation constraint rule library; running a multi-agent test, deploying a twin cluster in a virtual sandbox, and recording operation paths, attention heat maps, and decision delays; and generating a design defect map to locate cognitive friction points in the interface by comparing the deviation between the expected interaction flow and the actual behavior trajectory.

[0006] Core innovations include cognitive-behavioral twin modeling, semantic analysis of interactive environments, swarm intelligence testing and defect location.

[0007] Cognitive-Behavioral Twin Modeling: 1) Dynamic portrait generation: This integrates demographic attributes (age / occupation), cognitive preferences (risk-averse / exploratory), and behavioral history (operation trajectory) to inject typical cognitive biases (such as function neglect and interface misunderstanding) into the group through a generative adversarial network (GAN). 2) Psychological driving engine: Generates users' rationalized imaginations about unexperienced features based on the Large Language Model (LLM), for example: "Generation Z users may ignore security confirmation pop-ups in pursuit of efficiency."

[0008] Semantic analysis of interactive environment: 1) Encode interface elements into a machine-parseable semantic graph, including functional nodes (buttons / input boxes), logical edges (jump processes, dependencies), and physical constraints (mobile touch hotspots, VR gesture recognition thresholds). 2) Integrate environmental interference injectors (network delay, message push) to simulate real-world scenarios.

[0009] Swarm intelligence testing and defect location: 1) Supports concurrent testing of tens of thousands of digital humans, simulating multi-user resource competition (such as cursor preemption in a shared editing interface) through a conflict game sandbox; 2) Define the cognitive friction coefficient: comprehensive operation pause duration (>2s triggers an alert), path backoff times (>3 times to determine process redundancy), and pupil simulation focus deviation (line of sight deviates from the target element by more than 50px).

[0010] The technical effects are shown in Table 1: Table 2: Technical Effects index Traditional testing The present invention Testing cycle 3 weeks 48 hours Risk Forecasting Rate <60% 87% Special scenario coverage 20% 100%

[0011] The present invention can be widely applied to the following high-risk / high-cost / high-cycle testing scenarios of human-computer interaction systems: 1) Development of next-generation interactive hardware: Cognitive compatibility verification of brain-computer interface control devices (e.g., intelligent prostheses), holographic operating interfaces (e.g., surgical navigation systems), and sensory fusion terminals (e.g., digital olfactory devices); 2) Human Factors Engineering in Extreme Environments: reliability testing for special scenarios such as deep-sea operation robot control interfaces, space capsule human-machine collaborative systems, and disaster rescue equipment; 3) Swarm intelligence system optimization: digital avatar behavior prediction for the Metaverse social platform, drone swarm control systems, and multi-agent conflict resolution generation for shared collaboration tools; 4) Security testing in highly sensitive areas: Predicting cognitive friction in zero-tolerance error scenarios such as quantum computing consoles, medical device operating interfaces, and in-vehicle voice systems; 5) Development of universal interaction standards: Quantitative assessment of cross-cultural user cognitive biases in educational robots, emotional advertising systems, and consumer electronics.

[0012] Specifically, the application forms of the present invention include five forms: form a, b, c, d, and e.

[0013] Form a: Independent intelligent agent software system. Example adaptation: Quantum computing console (Example 11); 1) Operation mode: as an independent process communicating with a quantum simulator (such as Qiskit); 2) Output format: Generates a control button cognitive risk rating report (PDF / JSON); 3) Technical necessity: Exclusive system resources are required to run concurrent testing of tens of thousands of digital humans and cannot be dependent on design tools.

[0014] Form b: Embedded test firmware. Example adaptation: Brain-controlled intelligent prosthesis (Example 8); 1) Carrier: Factory self-test program of the prosthetic control chip (STM32H7 series); 2) Functionality: Operational reliability in the power-on automatic simulation of ALS patient breathing mode; 3) Technical necessity: Medical devices must meet IEC 62304 compliance and must have built-in traceable test modules.

[0015] Form c: SaaS platform service. Example adaptation: Emotion-aware advertising system (Example 12); 1) Operational flow: Advertising designer logs in to the platform → Uploads interface prototype → Selects user profile (teens / seniors) → Obtains emotional response heat map; 2) Commercial value: Charge based on the number of tests ($50 / 1,000 digital human tests), forming a subscription-based revenue model.

[0016] Form d: Hardware acceleration device. Example adaptation: Swarm drone control (Example 15); 1) Hardware carrier: Xilinx Versal ACAP chip; 2) Acceleration performance: 1,000 UAV collision prediction latency <5ms (CPU solution >200ms); 3) Technical barriers: The group game protocol (right 4) requires hardware to implement low-latency arbitration.

[0017] Form e: Mixed reality test suite. Example adaptation: Holographic medical navigation (Example 9); 1) Device composition: HoloLens 2 + haptic feedback gloves; 2) Testing process: The doctor wears the device and operates a virtual organ model, monitoring the gaze deviation alarm in real time; 3) Innovation: Cross-spatial interaction data fusion between virtual digital humans and physical gestures. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 System architecture diagram. This diagram shows a four-layer architecture: data collection layer (real user pool) → twin engine layer (GAN + LLM) → virtual environment layer (semantic parser + interference injector) → analysis layer (defect heatmap generation).

[0019] Figure 2 Cognitive-behavioral decision tree generation process. Demonstrates the reinforcement learning process from inputting a user profile to outputting N action paths, with emphasis on marking nodes where cognitive bias is injected.

[0020] Figure 3 An example of encoding a semantic graph for an interface. Taking an e-commerce app as an example, the "shopping cart-checkout-payment" process is converted into a weighted directed graph, with jump probability thresholds annotated.

[0021] Figure 4 A multi-agent conflict game sandbox. This example demonstrates a scenario where multiple digital humans collaborate to edit a document, showing a cursor grabbing probability model and conflict resolution protocol triggering logic.

[0022] Figure 5 : Principle of calculating cognitive friction coefficient. A timing diagram shows the algorithm that integrates three parameters: pause duration, sight offset, and path retraction. DETAILED DESCRIPTION

[0023] Example 1: Consumer electronics interface testing (mobile phone OS). Test objective: Verify the efficiency of seniors in operating the new voice assistant interface. 1) Twin construction: Input: 70+ years old, unfamiliar with touchscreens, prefers voice control; Cognitive bias injection: simulates "icon text that is too small increases the probability of accidental touch by 30%"; 2) Test results: The recognition rate of the "voice wake-up button" dropped by 60% in low-light environments. After optimization, the touch hotspot was increased.

[0024] Example 2: Accessible Interaction System Verification. Test Objective: Response reliability of the breath-controlled HMI system (Reference 5); 1) Environmental simulation: injecting abnormal respiratory rhythm data (such as the shallow breathing pattern of ALS patients); adding environmental interference (noise from ward equipment); 2) Output: The respiratory signal recognition delay is reduced from 1.2s to 0.3s, and the false trigger rate is reduced by 85%.

[0025] Example 3: Multi-user collaboration platform evaluation. Test target: Real-time collaboration functionality of Figma, an online design tool; 1) Group test: 200 digital humans were deployed for concurrent editing, triggering 173 "cursor grabbing conflicts"; 2) The game sandbox automatically generates solutions: automatically switches to partition editing mode when conflicts occur.

[0026] Example 4: In-vehicle voice system optimization. Test objective: Driving safety of navigation voice interaction; 1) Cognitive friction analysis: This simulates the probability of a driver's gaze being diverted from the road for more than 1 second while operating a voice interface on a highway. 2) Optimization solution: Split complex instructions into single responses, reducing the duration of a single interaction by 40%.

[0027] Example 5: Medical device interface verification. Test objective: Risk of misoperation in the MRI operating interface; 1) Special scenario coverage: Simulates the pressure error when doctors wearing gloves operate touch screens; generates simplified operation paths in emergency situations (such as patient convulsions); 2) Output: The number of high-risk points for misoperation was reduced from 12 to 2.

[0028] Example 6: Child Interaction Testing of Educational Robots. Scenario Pain Point: Children's educational robots need to adapt to the cognitive differences of children aged 3-8. Traditional tests cannot cover special behaviors such as attentional blindness and symbol misunderstanding. 1) Patented technology application. Build a library of children's cognitive biases: Instill a tendency to visualize abstract symbols (e.g., associating the mathematical symbol ∞ with a slide); Physiological parameter compensation: Monitor and simulate the attention decay curve of children with ADHD (average focus duration <8s); 2) Test results: It was found that the robot screen flickering frequency conflicted with the threshold for inducing epilepsy in children. After optimization, it passed ISO8124 safety certification.

[0029] Example 7: Metaverse Digital Avatar Social Testing. Scenario Pain Point: The Metaverse social platform needs to predict the authenticity of the digital avatar's behavior to avoid the "uncanny valley effect"; 1) Patented technology application. Social network simulation: embedding intimacy influencing factors (for every +1 level of friend intimacy, message reply delays are reduced by 0.3s); group game protocol: simulating gift reward competition scenarios to trigger resource preemption strategies; 2) Test Results: Optimizing the digital avatar’s micro-expression response delay reduced the incidence of the uncanny valley effect by 92%. Example 8: Interactive Verification of Brain-Controlled Intelligent Prosthesis. Scenario Pain Points: Brain-computer interface prostheses need to maintain operational stability in complex environments, and existing tests lack simulation of physiological signal interference. 1) Patented technology application: EEG semantic graph encoding: Setting the EMG noise filtering threshold (>200μV to initiate filtering); Cognitive friction calculation: Incorporating HRV parameters (operation accuracy drops by 40% when heart rate variability >0.1); 2) Test results: In a simulated subway vibration environment, the grasping success rate increased from 68% to 95%. Example 9: Holographic medical navigation system testing. Scenario pain points: Holographic surgical navigation requires ensuring the accuracy of doctors' operations under high pressure, and extreme scenarios are difficult to test physically. 1) Patented technology application. Device parameter library call: Set the holographic projection brightness compensation algorithm in a blood mist environment; Dynamic friction coefficient: Trigger an alarm when the operator's line of sight deflects >5° for 2 seconds; 2) Test results: The cognitive load of key operating steps was reduced by 43%, and the surgical error rate dropped by 70%.

[0030] Example 10: Space capsule human-machine collaborative system. Scenario pain point: Deep space exploration missions require verification of the collaborative efficiency of multiple astronauts and AI in a weightless environment. 1) Patented technology application. Swarm Intelligence Sandbox: simulates oxygen competition scenarios within the cabin, triggering a competitive and cooperative strategy tree; Cross-cultural cognitive adaptation: automatically adjusts the decision-making hesitation thresholds of astronauts of different nationalities; 2) Test Results: The emergency protocol response process was optimized, and mission completion time was reduced to 120% of ground simulation time.

[0031] Example 11: Quantum Computing Control Interface Verification. Scenario Pain Point: The quantum computer control console needs to accommodate the cognitive differences between physicists and engineers. Traditional testing cannot simulate the cognitive load of the quantum state visualization interface. 1) Patented technology application. Build a library of professional cognitive biases: physicists prefer wave function diagrams (85% confidence), engineers prefer bitstream values ​​(90% confidence); environmental interference simulation: the impact of injected quantum decoherence noise on interface stability; 2) Test results: The cognitive friction coefficient of the quantum gate operation button was found to be 0.9 (very dangerous), and the error rate decreased by 92% after optimization.

[0032] Example 12: Testing an Emotion-Aware Advertising System. Scenario Pain Point: Emotional AI billboards need to predict the emotional responses of different groups of people to avoid triggering a chain reaction of negative emotions. 1) Patented technology application. Physiological signal fusion: Emotional response models are constructed by simulating pupil dilation rate (excitement) and galvanic skin response (disgust); Group behavior prediction: Emotional contagion pathways are simulated in a plaza (one person's negative emotions trigger a 15% group avoidance rate); 2) Test results: After system calculations and optimization of the content push algorithm, the negative emotion triggering rate dropped from 23% to 3%.

[0033] Example 13: Deep-sea exploration robot HMI verification. Scenario pain point: The deep-sea robot control interface needs to maintain operational reliability in a high-pressure environment, and the physical testing cost is extremely high. 1) Patented technology application. Extreme environment modeling: The impact coefficient of pressure injection at a depth of 1,000 meters on operating accuracy (hand tremor amplitude + 40%); cognitive friction warning: Automatically switch to a simplified interface when the operation pauses for more than 5 seconds; 2) Test results: The success rate of key robotic arm operations increased from 74% to 98%.

[0034] Example 14: Digital Olfactory Interaction System Testing. Scenario Pain Points: The new generation of odor interaction devices lacks mature testing methods, and the odor-behavior association model has not yet been established. 1) Patented technology application. Novel semantic graph encoding: Establishes mapping rules between odor molecular structure and emotional response (fruity = pleasure, rancid = avoidance); Cognitive bias injection: Simulates the sensitivity decay curve caused by olfactory fatigue (30% decrease every 5 minutes) 2) Test results: The first digital scent interaction standard was established, with user expected compliance reaching 89%.

[0035] Example 15: Swarm UAV Control Interface. Scenario Pain Point: Controlling a swarm of hundreds of drones requires predicting the operator's cognitive overload risk. Traditional methods cannot simulate emergency obstacle avoidance scenarios. 1) Patented technology application. Multi-agent game protocol: autonomous negotiation algorithm for simulating drone conflicts; cognitive load monitoring: triggering hierarchical display when interface information density exceeds 50 elements / screen; 2) Test results: System simulation and calculation showed that emergency response speed increased by 3 times and the operational error rate decreased by 87%.

Claims

1. A virtual testing method and device for an interactive system based on multimodal cognitive twins, characterized in that The following technical closed loops must be included: a) Cognitive twin construction: (1) Input the demographic attribute data (age / occupation / cultural background), cognitive characteristic data (risk preference / attention decay curve), and behavioral history data (operation trajectory log) of the target user group. (2) Implant typical cognitive biases of the group (including function neglect rate, interface misreading probability, and decision hesitation threshold) through the anti-bias injection module. (3) Output an N-branch decision tree (N ≥ 5) with behavior confidence intervals; b) Cross-modal interaction environment adaptation: (1) Parse the interactive system to be tested into a machine-executable semantic graph, including: (2) Functional nodes (physical buttons, voice command area, brain-computer interface sensing area) (3) Logical constraints (state jump rules, multimodal input priority) (4) Device physical parameters (VR controller damping coefficient, EEG signal sampling rate threshold); c) Swarm Intelligence Sandbox Testing: (1) Deploy ≥ 1000 cognitive twins to operate concurrently in a virtual environment, (2) Real-time recording of multi-dimensional behavioral fingerprints (operation path entropy, pupil simulation focus coordinates, and electromyography simulation feedback intensity); d) Quantitative positioning of cognitive friction: (1) Calculate the dynamic friction coefficient = F (operation pause duration, number of path retractions, target element recognition deviation rate), (2) When the coefficient exceeds the industry safety threshold (such as medical equipment > 0.7, consumer electronics > 0.4), the interface defect topology map is automatically output.

2. As described in right 1, it is characterized in that The cognitive bias injection module must implement: (1) Adjust bias parameters based on the cross-cultural cognitive difference database (e.g., East Asian users’ pop-up window ignoring rate +15%), (2) Generate cognitive adversarial examples (e.g., deliberately confuse icons with similar functions) through generative adversarial networks (GANs).

3. As described in right 1, it is characterized in that A machine-executable semantic graph must contain: (1) New interactive device interfaces: EEG signal decoding rules for brain-computer interfaces, eye tracking tolerance intervals for AR glasses, (2) Environmental interference factor library: 5G network fluctuation model, public space noise spectrum.

4. As described in right 1, it is characterized in that Swarm intelligence sandbox testing must support: (1) Social network simulation: In the social product test, factors affecting friend intimacy (such as message reply delay attenuation coefficient) are embedded. (2) Resource conflict game protocol: When multiple users compete for limited resources (such as shared computing power), a competitive strategy tree (auction / polling / preemption) is triggered.

5. As described in right 1, it is characterized in that The dynamic friction coefficient calculation must incorporate: (1) Physiological signal simulation value: the weight of the impact of heart rate variability (HRV) on operation accuracy under stress scenarios, (2) Cognitive load index: the mismatch between the information density of the interface and the user’s working memory capacity.

6. A carrier or device for implementing the method according to claim 1, characterized in that It can take one or more of the following forms: a) Independent agent software system: deployed on a cloud computing platform, providing an API interface for interactive design tools (such as Figma / Adobe XD) to call testing services; b) Embedded test firmware: Factory test modules integrated into human-computer interaction hardware devices (such as car center consoles and surgical robot control panels) to perform device-level cognitive safety verification; c) SaaS platform service: Configure test parameters through the web page and generate multimodal test reports (including cognitive friction heat map / behavior trajectory video); d) Hardware Accelerator: This device accelerates group behavior simulation based on FPGA chips, specifically for scenarios with real-time requirements of >100fps, such as autonomous driving systems and drone swarms. e) Mixed Reality Test Suite: Combined with AR glasses to project a virtual interface, digital human operation feedback is superimposed on the physical product prototype.

Images