Test scenario recovery method, head-mounted display device, and computer-readable medium

Abstract

Embodiments of the present disclosure disclose a test scene recovery method, a head-mounted display device and a computer readable medium, comprising: in response to detecting a test case recording operation, and the interaction operation satisfying a preset scene snapshot generation condition, generating scene snapshot information of a corresponding test scene according to object space information and object interaction state information of a current test scene; performing associated storage processing on the scene snapshot information and the interaction operation; in response to detecting a test case playback operation, and the interaction operation to be executed satisfying the preset scene snapshot generation condition, performing recovery processing on the test scene according to the scene snapshot information corresponding to the interaction operation to be executed. The embodiment can improve the test effect, and reduce the calculation resources and time consumption consumed by positioning error causes.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of automated testing, specifically to a test scenario recovery method, a head-mounted display device, and a computer-readable medium. Background Technology

[0002] When testing AR or VR scenes, the current scene can be conveniently tested by reproducing the test scene. However, the synchronization sequence of data within the test scene also affects the effectiveness and accuracy of the test. Currently, common methods for testing AR or VR scenes include script-based or recorded operation playback, testing based on a fixed initial scene, or log-based or time-triggered testing.

[0003] However, in practice, it has been found that the following technical problems often arise when using the commonly used methods for testing: When using operation replay, only the user's actions are replayed without reproducing the spatial state of the test scenario at the time of the user's actions. This results in different results for the same test case in different executions, leading to poor test performance. When testing based on the above commonly used methods, subtle changes in the spatial position, orientation, parent-child hierarchy, and relative relationships of virtual objects in AR / VR scenes result in poor consistency of spatial relationships, leading to poor test performance. When tests fail, the lack of complete scene state records makes it difficult to reproduce the real environment when the problem occurred, making it difficult to locate the cause of the error, consuming more computing resources, and taking longer.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide a test scene recovery method, a head-mounted display device, and a computer-readable medium to address one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a test scenario recovery method, including: in response to detecting a test case recording operation and the interaction operation meeting preset scenario snapshot generation conditions, generating scenario snapshot information corresponding to the test scenario based on the object space information and object interaction state information of the current test scenario; performing associated storage processing on the scenario snapshot information and the interaction operation; in response to detecting a test case playback operation and the interaction operation to be executed meeting preset test recovery conditions, performing test scenario recovery processing based on the scenario snapshot information corresponding to the interaction operation to be executed.

[0008] Optionally, the method further includes: collecting object space information of the current test scenario; and collecting interaction data collection information of the current test scenario.

[0009] Optionally, the method further includes storing the above-mentioned scene snapshot information and the above-mentioned interactive operations into an associated database.

[0010] Optionally, the method further includes: obtaining scene snapshot information corresponding to the interactive operation to be executed from the associated database; loading the scene snapshot information; and restoring the test scene based on the loaded scene snapshot information.

[0011] Optionally, the method further includes: performing format conversion processing on the above-mentioned object space information and the above-mentioned object interaction state information to obtain formatted object space information and object interaction state information; and determining the scene snapshot information of the corresponding test scenario based on the above-mentioned formatted object space information and object interaction state information.

[0012] In a second aspect, some embodiments of this disclosure provide a head-mounted display device, including: one or more processors; a storage device for storing one or more programs; an optomechanical system and optical elements for imaging in front of a user; and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementations of the first aspect.

[0013] Thirdly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0014] The above embodiments of this disclosure have the following beneficial effects: A test scene recovery method according to some embodiments of this disclosure can improve test performance and reduce the computational resources and time consumed in locating the cause of errors. The reasons for poor test performance and high computational resources and time consumption in locating the cause of errors are as follows: When using operation replay, only user operations are replayed, without reproducing the spatial state of the test scene at the time of user operation. This leads to different results for the same test case in different executions, resulting in poor test performance; when testing based on the commonly used methods, subtle changes in the spatial position, orientation, parent-child hierarchy, and relative relationships of virtual objects in AR / VR scenes result in poor consistency of spatial relationships, leading to poor test performance; when a test fails, the lack of complete scene state records makes it difficult to reproduce the real environment at the time of the problem, making it difficult to locate the cause of errors, consuming more computational resources and time. Based on this, some embodiments of this disclosure provide a test scenario recovery method, including: First, in response to detecting a test case recording operation and the interaction operation meeting preset scenario snapshot generation conditions, generating scenario snapshot information corresponding to the test scenario based on the object space information and object interaction state information of the current test scenario. This allows the generation of a scenario snapshot of the current test scenario for recording the current test scenario. Then, the scenario snapshot information and the interaction operation are associated and stored. This allows the recording of the scenario snapshot generated in the current test scenario and the corresponding interaction operation. Finally, in response to detecting a test case playback operation and the interaction operation to be executed meeting preset test recovery conditions, the test scenario is recovered based on the scenario snapshot information corresponding to the interaction operation to be executed. This allows the test scenario to be recovered for error localization. Because it does not merely play back user operations, but records the test scenario by generating a scenario snapshot of the current test scenario, the spatial state of the test scenario during user operations can be reproduced, thereby improving the testing effect. Furthermore, because the scene snapshots are generated based on object spatial information and object interaction state information, the consistency of spatial relationships in AR / VR scenes can be improved. This, in turn, enhances testing effectiveness. Moreover, since the test environment can be reproduced through the generated scene snapshot information, and this information can accurately represent the real-world environment where problems occurred during testing, the difficulty of locating the cause of errors is reduced, thereby decreasing the computational resources consumed in error localization and shortening the testing time. Attached Figure Description

[0015] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0016] Figure 1 This is an exemplary system architecture diagram of a test scene recovery method, a head-mounted display device, and a computer-readable medium according to some embodiments of this disclosure; Figure 2 This is a flowchart of some embodiments of the test scene recovery method, head-mounted display device, and computer-readable medium according to the present disclosure; Figure 3 This is a flowchart of another embodiment of the test scene recovery method, head-mounted display device, and computer-readable medium according to the present disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0019] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0020] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0021] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0022] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] Figure 1 An exemplary system architecture 100 is shown that can be applied to embodiments of the test scene recovery method, head-mounted display device, and computer-readable medium disclosed herein.

[0024] like Figure 1 As shown, the exemplary system architecture 100 may include a head-mounted display device 11 and a terminal device 12.

[0025] The head-mounted display device 11 may include one or two optomechanical and optical elements 111. These optomechanical and optical elements are used to display a virtual interface. Furthermore, the head-mounted display device 11 also includes a frame 112. In some embodiments, the sensors, processing unit, memory, and battery of the head-mounted display device 11 may be housed within the frame 112. In some alternative implementations, one or more components of the sensors, processing unit, memory, and battery may be integrated into another separate accessory (not shown), connected to the frame 112 via a data cable 13. In some alternative implementations, the head-mounted display device 11 may only have display functionality and some sensors, while providing data processing, data storage, power supply, and other capabilities through a terminal device 12.

[0026] The terminal device 12 includes a touch component 121, which can be a touch screen or a touchpad. The touch screen can be used to display content and control window dragging, and the touchpad can be used to control window dragging. In some embodiments, the head-mounted display device 11 and the terminal device 12 can be communicatively connected. The communication connection can be a wired connection or a Bluetooth connection.

[0027] It should be understood that Figure 1 The number of head-mounted display devices and terminal devices shown is merely illustrative. Any suitable number of head-mounted display devices and terminal devices can be included depending on the implementation requirements.

[0028] Further reference Figure 2 The diagram illustrates a flow 200 of some embodiments of a test scenario recovery method according to the present disclosure. The test scenario recovery method includes the following steps: Step 201: In response to the detection of a test case recording operation and the interaction operation meeting the preset scenario snapshot generation conditions, generate the corresponding test scenario snapshot information based on the object space information and object interaction state information of the current test scenario.

[0029] In some embodiments, in response to detecting a test case recording operation and the interaction operation meeting preset scene snapshot generation conditions, the execution entity can generate scene snapshot information for the corresponding test scenario based on the object space information and object interaction state information of the current test scenario. The scene snapshot information can be a snapshot file of the generated current test scenario. The test case recording operation can be an operation where a user clicks a control to start recording a test case. For example, when executing a test case for the first time, clicking the control to start recording the test case generates a scene snapshot related to the test case, which can be used to restore the test scenario when the test case is executed again. The test cases can be used to test the hardware, engine, or business functions of the head-mounted display device. The specific test content for testing the hardware, engine, or business functions of the head-mounted display device is not limited. For example, it can be used to test the hardware performance of the camera and IMU of the head-mounted display device; it can be used to test the running effect of the SLAM positioning engine, rendering engine, and gesture recognition of the head-mounted display device; and it can be used to test the business functions of the UI operation, virtual-real overlay, and spatial anchoring of the head-mounted display device. The aforementioned preset scene snapshot generation condition can be that the aforementioned interactive operation is the same as one of the preset execution operations in the preset execution operation set. The aforementioned interactive operation can be the operation corresponding to a single operation node in the aforementioned test case that has been completed at the current moment. The aforementioned operation node can be the node where the interactive operation is located. For example, if the operation node is a spatial anchor point creation node, then the interactive operation can be a spatial anchor point creation operation. The preset execution operations in the aforementioned preset execution operation set can represent the interactive operations in the aforementioned test case that need to generate scene snapshot information in a pre-defined manner. For example, the preset execution operations in the preset execution operation set can be a 3D model drag-and-drop operation or a spatial anchor point creation operation. The aforementioned current test scene can be the scene in which the aforementioned head-mounted display device is running the aforementioned test case at the current moment. For example, the aforementioned current test scene can be composed of the screen rendered by the head-mounted display device, object space information, and object interaction state information. There is no limitation on the specific type of the aforementioned head-mounted display device; for example, the head-mounted display device can be AR glasses. The aforementioned object space information can be the pose of each virtual object in the current test scene. The aforementioned object interaction state information can be the device state in the current test scene, the collected user interaction information, and the triggered interaction events. The aforementioned device status may include, but is not limited to, one of the following: whether the head-mounted display device is connected, whether it is powered on, the controller offset, the remaining battery power of the head-mounted display device, and the sampling frame rate of the head-mounted display device in the current scene. The aforementioned interaction information may include gaze interaction information, gesture interaction information, and controller interaction information. The aforementioned gaze interaction information may include the ray projected by the user's gaze, the gaze point, and the gaze duration.The ray projected by the aforementioned gaze can be a spatial ray formed by the user's gaze through AR glasses. The aforementioned gaze point can be the virtual object that the projected spatial ray collides with, and the 3D coordinates of the collision. The aforementioned gaze duration can be the length of time the user gazes at a virtual object. The aforementioned gesture interaction information can be the user's gestures. The specific type of the aforementioned gesture interaction information is not limited and can be set according to actual needs. For example, gesture interaction information can be an open palm, a clenched fist, or a single-finger tap. The aforementioned controller interaction information can be the user's actions on the controller. The specific type of the aforementioned controller is not limited and can be set according to actual needs; for example, the controller can be a 6DoF gamepad. The aforementioned triggered interaction events can be interaction events triggered by the user's actions. For example, an interaction event can be a pinching and closing of the thumb and forefinger, triggering the event of picking up a virtual object.

[0030] In some optional implementations of certain embodiments, in response to detecting a test case recording operation and the interaction operation meeting preset scenario snapshot generation conditions, the aforementioned execution entity can generate scenario snapshot information for the corresponding test scenario based on the object space information and object interaction state information of the current test scenario through the following steps: The first step is to perform format conversion processing on the aforementioned object space information and object interaction state information to obtain formatted object space information and object interaction state information. In practice, the aforementioned execution entity can use a JSONSchema format conversion algorithm to perform format conversion processing on the aforementioned object space information and object interaction state information to obtain formatted object space information and object interaction state information.

[0031] The second step is to determine the scene snapshot information for the corresponding test scenario based on the formatted object space information and object interaction state information. In practice, firstly, the executing entity can use the preset generated scene snapshot information number, the UTC timestamp obtained from the system UTC clock, and the ID of the aforementioned interaction operation as metadata. The ID of the aforementioned interaction operation can be the number of the interaction operation. For example, the number can be 02. Then, the formatted object space information, object interaction state information, and the determined metadata are used to determine the scene snapshot information for the corresponding test scenario.

[0032] Optionally, in response to the detection of a test case recording operation, and if the interaction operation meets the preset scenario snapshot generation conditions, the aforementioned execution entity may also perform the following steps: The first step is to collect the object space information of the current test scenario. In practice, the aforementioned execution entity can call a data collection interface to obtain the object space information of the current test scenario. This data collection interface can be an API interface used to obtain both the object space information of the current test scenario and interactive data collection information.

[0033] Alternatively, the aforementioned execution entity can also obtain the object space information of the current test scene by calling the data acquisition interface through the Unity engine.

[0034] The second step is to collect interaction data information for the current test scenario. In practice, the aforementioned execution entity can call the data collection interface to obtain the interaction data information for the current test scenario.

[0035] Optionally, the aforementioned execution entity can also use the Unity engine to call the aforementioned data acquisition interface to obtain interactive data acquisition information of the current test scene.

[0036] Step 202: Perform associated storage processing on scene snapshot information and interactive operations.

[0037] In some embodiments, the executing entity may perform associated storage processing on the scene snapshot information and the interaction operations. In practice, the executing entity may store the scene snapshot information and the interaction operations in an associated database.

[0038] Step 203: In response to the detection of a test case replay operation and the interaction operation to be executed meeting the preset test recovery conditions, the test scenario is restored according to the scene snapshot information of the corresponding interaction operation to be executed.

[0039] In some embodiments, the execution entity may, in response to detecting a replay operation of a test case and finding that the interaction to be executed meets preset test recovery conditions, restore the test scene based on the scene snapshot information corresponding to the interaction to be executed. The preset test recovery conditions may be scene snapshot information corresponding to the interaction to be executed stored in an associated database. The restored test scene may be generated from the scene snapshot information of the interaction to be executed, and may be the scene restored when the head-mounted display device replays the test case and executes the interaction to be executed, including the image rendered by the head-mounted display device, object space information, and object interaction state information. The replay operation of the test case may be an operation where a user clicks a control that generates the test case with scene snapshot information to execute again. The aforementioned interactive operations to be executed can be the interactive operations that need to be executed at the current moment, or the operations corresponding to a single operation node to be executed in the currently replayed test case. If the interactive operations to be executed meet the above-mentioned preset test recovery conditions, it means that when the test case currently being replayed is executed for the first time, the scene snapshot information of the corresponding interactive operations to be executed has been generated. Therefore, during the replay of the test case, when the interactive operations to be executed need to be executed, it is only necessary to load the corresponding scene snapshot information to restore the corresponding test scene.

[0040] In addressing the technical challenges mentioned above, the application scenario—testing industrial AR equipment maintenance and assembly guidance—often presents the following issues: When logs are printed independently for each pipeline, and data is roughly matched using only the system's timestamps, the timing synchronization determination results are distorted, leading to poor test performance. This necessitates repeated execution of equipment maintenance and assembly guidance operations, consuming significant computational resources and time. Considering the specific requirements of this application scenario—high timing accuracy in equipment assembly—we have decided to adopt the following solution: Optionally, the aforementioned implementing entity may also perform the following steps: The first step is to determine the current time anchor point as the target reference anchor point in response to the detected trigger operation of the operation node. This current time anchor point can be the timestamp of the CPU of the aforementioned head-mounted display device at the current moment. In practice, the executing entity can determine the timestamp of the CPU of the aforementioned head-mounted display device at the current moment as the target reference anchor point.

[0041] The second step is to construct a symmetrical timing sampling window based on the aforementioned target reference anchor point. This symmetrical timing sampling window can be a sampling window composed of a pre-sampling window and a post-sampling window centered on the target reference anchor point. In practice, the executing entity can use a target number of frames preceding the target reference anchor point, a target number of frames following the target reference anchor point, and one frame corresponding to the target reference anchor point to form the symmetrical timing sampling window. The specific number of target frames is not limited and can be set according to actual needs. For example, if the target number is 30 frames, the symmetrical timing sampling window can consist of one frame corresponding to the target reference anchor point, 30 frames preceding the target reference anchor point, and 30 frames following the target reference anchor point.

[0042] The third step involves collecting rendering pipeline data, physics pipeline data, and skeletal animation pipeline data for each virtual object in the current scene, based on the aforementioned symmetrical temporal sampling window. This results in rendering pipeline data sets, physics pipeline data sets, and skeletal animation pipeline data sets. The rendering pipeline data can be the data used when the GPU rendering pipeline draws the virtual objects. This data can include the MVP matrix, rendering level, rendering markers, screen rendering coordinates, and LOD level. The rendering markers can be the identifier metadata of the virtual objects. For example, the rendering marker can be 1. The screen rendering coordinates can be the two-dimensional coordinates of the virtual object's vertices on the window screen after coordinate transformation. The rendering level can be the sequence number assigned to the virtual objects, used to distinguish the order in which they are drawn. The physics pipeline data can be the pose, linear velocity, angular velocity, ground contact state, and touch state of the virtual objects in the world coordinate system. The ground contact state indicates whether the virtual object is in contact with the ground plane. The touch state indicates whether the virtual object collides with other virtual objects or other geometric surfaces. The aforementioned geometric surfaces can be the geometric surfaces of the current real-world scene. The aforementioned skeletal animation pipeline data can be the global pose of the skeletons, the local transformation matrices of the sub-bones, and the activation states of the bone hierarchy of the aforementioned virtual object. The aforementioned global pose of the skeletons can be the spatial position and rotational orientation of each bone in the virtual object in the world coordinate system. The aforementioned local transformation matrices of the sub-bones can be 4th-order homogeneous transformation matrices corresponding to the translation, rotation, and scaling of the sub-bone relative to the parent bone, using the parent bone's coordinate system as a reference. The aforementioned activation states of the bone hierarchy can be state parameters indicating whether each bone in the virtual object is active. For example, a state parameter of 1 indicates that the bone has been activated; a state parameter of 0 indicates that the bone has not been activated. It should be noted that when the aforementioned state parameters indicate that the bone is not activated, its posture is fixed and no longer changes with the animation. The aforementioned virtual object can include at least one parent bone and one child bone. In practice, the aforementioned execution entities can collect rendering pipeline data for each virtual object through the Renderer and Camera components within the aforementioned symmetrical timing sampling window; collect physical pipeline data for each virtual object using the Rigidbody and Collider components; and collect skeletal animation pipeline data for each virtual object using the Animator's built-in animation frame callback method.

[0043] The fourth step involves aligning the rendering pipeline data group, the physical pipeline data group, and the skeletal animation pipeline data group according to the aforementioned target reference anchor point. In practice, the execution entity can use a nearest neighbor matching algorithm to align these data groups based on the target reference anchor point.

[0044] The fifth step involves denoising the multi-frame data information included in the aforementioned symmetrical temporal sampling window to obtain denoised multi-frame data information. In practice, the aforementioned execution entity can use a moving average filtering algorithm to denoise the multi-frame data information included in the aforementioned symmetrical temporal sampling window.

[0045] The sixth step is to split each virtual object in the current scene to obtain the split virtual objects. In practice, the above-mentioned execution entity can use a hash grouping and aggregation algorithm to split each virtual object in the current scene to obtain the split virtual objects.

[0046] The seventh step involves defining the physical compensation coefficients, interpolation and denoising parameters, and skeletal motion constraint parameters as snapshot metadata. The skeletal motion constraint parameters can include bone rotation limits and IK constraints. In practice, the execution entity can call the engine's built-in global static interface to obtain the physical compensation coefficients, interpolation and denoising parameters, and skeletal motion constraint parameters. Then, these obtained parameters are defined as snapshot metadata.

[0047] Step 8 involves packaging the snapshot metadata, virtual object attribute information, and the normalized rendering pipeline data set, physical pipeline data set, and skeletal animation pipeline data set into a hierarchical structure. The virtual object attribute information may include a unique identifier, geometric dimensions, and bounding box for each virtual object. In practice, a serialization encapsulation algorithm can be used to package the snapshot metadata, virtual object attribute information, and the normalized rendering pipeline data set, physical pipeline data set, and skeletal animation pipeline data set into a hierarchical structure.

[0048] The ninth step is to perform validation processing on the packaged data to obtain the validated data. In practice, the aforementioned executing entity can use MD5 to validate the packaged data to obtain the validated data.

[0049] Step 10: Add the validated data to the object space information.

[0050] The above technical solution, as an inventive point of this disclosure, solves the technical problem of "distorted timing synchronization determination results, leading to poor test performance, requiring repeated execution of equipment maintenance and assembly instructions for testing, consuming significant computing resources and time." The reasons for this distortion, poor test performance, and the need for repeated execution of equipment maintenance and assembly instructions for testing, resulting in significant computing resource consumption and time consumption, are as follows: When logging each pipeline independently and relying solely on the time system timestamps for rough data matching, the timing synchronization determination results become distorted, leading to poor test performance, requiring repeated execution of equipment maintenance and assembly instructions for testing, consuming significant computing resources and time. Therefore, this disclosure adopts the following approach: First, a symmetrical sampling window is built using the CPU timestamp triggered by interaction as a unified reference anchor point. The collected rendering pipeline data group, the aforementioned physical pipeline data group, and the aforementioned skeletal animation pipeline data group are aligned synchronously. This achieves timing alignment processing, and also performs sliding mean denoising and verification, thereby improving the reliability of the data and the consistency of timing. This can improve the testing effect, thereby reducing the need to repeatedly execute equipment maintenance and assembly guidance operations for testing, consuming computing resources, and shortening the time consumption.

[0051] In addressing the aforementioned technical issues by adopting technical solutions, the following technical problems often arise in the application scenario: compliance verification testing of SOPs (Standard Operating Procedures). Compliance verification through step-by-step comparison of recorded logs is time-consuming and only allows comparison of step names, failing to verify the overall topological logic of the process, resulting in poor testing effectiveness and long processing times. Considering the following requirements for this application scenario: SOPs demand high accuracy in execution steps, we have decided to adopt the following solution: Optionally, the aforementioned implementing entity may also perform the following steps: The first step is to obtain the execution step information for each step in the interactive operation corresponding to the current scenario. The execution step information can be the steps involved in performing the interactive operation.

[0052] The second step involves standardizing the execution step information to obtain an execution step information sequence. In practice, firstly, the executing entity can sort the execution step information in ascending order based on the timestamp corresponding to each execution step, resulting in a sorted execution step information sequence. Then, a moving average filtering algorithm can be used to denoise the execution step information sequence, yielding a denoised execution step information sequence.

[0053] The third step involves performing redundancy detection on the aforementioned sequence of execution steps to obtain redundancy detection results. These results can indicate whether redundant execution steps exist within the sequence. In practice, the executing entity can determine the hash value of each execution step in the sequence. Then, in response to the determination that the obtained hash values ​​contain the same hash value, the redundant execution steps are identified as redundancy detection results. Conversely, in response to the determination that the obtained hash values ​​do not contain the same hash value, the non-redundant execution steps are identified as redundancy detection results.

[0054] Fourth, in response to the determination that the above redundancy detection result information meets the preset redundancy condition, the above execution step information is updated to obtain an updated execution step information sequence. The preset redundancy condition can be the redundancy detection result information indicating the existence of redundant execution step information in the execution step information sequence. In practice, the executing entity can retain one of the execution step information pieces with the same hash value in the above execution step information sequence and delete the remaining execution step information pieces to update the above execution step information sequence, obtaining the updated execution step information sequence.

[0055] The fifth step is to generate an operation step graph based on a preset set of interactive operation information. This operation step graph can be a graph of the constructed preset set of interactive operation information. The preset interactive operation information in this set can be pre-defined interactive operations to be performed. In practice, the executing entity can generate the operation step graph based on the preset set of interactive operation information using a DAG (Directed Acyclic Graph) construction algorithm.

[0056] Step 6: Based on the aforementioned operation step graph, determine the operation matching link information for each execution step in the updated execution step information sequence. This operation matching link information can be the execution path of the execution step information. The execution path can include the node in the aforementioned operation step graph corresponding to the execution step information, the preset interactive operation information to be executed before the execution step information, and the preset interactive operation information to be executed after the execution step information. In practice, the executing entity can use a Breadth-First Search (BFS) matching algorithm to match each execution step in the updated execution step information sequence with the aforementioned operation step graph to obtain the corresponding operation matching link information for that execution step information.

[0057] The seventh step is to map each execution step information in the updated execution step information sequence to a topology node of the topology network to be constructed, thereby obtaining each topology node, and to bind each execution step information to the corresponding topology node.

[0058] Step 8: Based on the step dependencies, step jumps, and cross-level constraints between the various topological nodes, generate the directed edges of the topological network to be constructed. The step dependencies can be dependencies between the various topological node groups. For example, a topological node group can require the previous topological node group to execute before it can execute. Step jumps can require the current topological node group to execute before jumping to the next topological node group. Cross-level constraints can prevent two non-adjacent topological node groups from executing sequentially. In practice, the execution entity can use a rule-matching traversal algorithm to generate the directed edges of the topological network to be constructed based on the step dependencies, step jumps, and cross-level constraints between the various topological node groups.

[0059] Step 9: Construct a global temporal topological adjacency matrix based on each topological node and the directed edges of the topological network. This global temporal topological adjacency matrix can be a two-dimensional square matrix representing the connectivity relationships between topological nodes. The matrix row and column indices correspond one-to-one with the topological node numbers. A cell is assigned a value of 1 if there is a directed edge between two topological nodes, and a value of 0 if there is no connectivity. In practice, an equal-order, all-zero square matrix is ​​initialized according to the total number of topological nodes. All directed edges are traversed, and the matrix positions corresponding to the start and end nodes of the directed edges are assigned a value of 1. Then, the Kahn topological sorting algorithm is used to verify that there are no cycles in the topological structure, thus obtaining the constructed global temporal topological adjacency matrix.

[0060] Step 10: Match the aforementioned global temporal topological adjacency matrix and the aforementioned operation step graph to obtain various detection result information. In practice, firstly, a global temporal topological adjacency matrix can be constructed based on the aforementioned operation step graph as a verification benchmark. Then, the Ullmann subgraph matching algorithm is used to compare whether the matrices transformed from the aforementioned global temporal topological adjacency matrix and the aforementioned operation step graph are consistent. Next, the feature vector of each execution step information in the aforementioned execution step information sequence is determined. Then, the preset standard step information corresponding to the aforementioned execution step information in the preset standard step information sequence is determined as the target preset standard step information. The preset standard step information in the aforementioned preset standard step information sequence can be the standard execution steps corresponding to the aforementioned execution step information. Secondly, the cosine similarity algorithm is used to determine the similarity between the feature vector of the target preset standard step information and the feature vector of the aforementioned execution step information. Then, it is determined whether the obtained similarity is less than a preset similarity. There is no specific limitation on the aforementioned preset similarity; it can be set according to actual needs. Finally, in response to the determination that the similarity between the target preset standard step information and the above execution step information is less than the preset similarity, or that the matrix transformed from the global temporal topological adjacency matrix and the operation step graph is inconsistent, the detection error is determined as the detection result information corresponding to the above execution step information. In response to the determination that the similarity between the target preset standard step information and the above execution step information is greater than or equal to the preset similarity, and that the matrix transformed from the global temporal topological adjacency matrix and the operation step graph is the same, the detection success is determined as the detection result information corresponding to the above execution step information.

[0061] The eleventh step is to store the obtained detection results into scene snapshot information.

[0062] The above-described technical solution, as an inventive point of this disclosure, solves the technical problem of "the time-consuming comparison process, which only allows for comparison of step names and fails to verify the overall topological logic of the process, resulting in poor test performance and long testing time." The reasons for this are as follows: When logs are printed independently for each pipeline, and data is roughly matched using only the time system timestamps, the timing synchronization judgment results are distorted, leading to poor test performance. This necessitates repeated execution of equipment maintenance and assembly instructions for testing, consuming significant computational resources and time. Therefore, this disclosure employs the following method: collecting all practical operation steps, sorting and filtering them for noise reduction, and generating a topological network of operation steps. This network is then compared with a standard operation step graph to verify the operation steps. Therefore, it is not necessary to verify only the step names, thereby improving the accuracy of verification and the test performance.

[0063] The above embodiments of this disclosure have the following beneficial effects: A test scene recovery method according to some embodiments of this disclosure can improve test performance and reduce the computational resources and time consumed in locating the cause of errors. The reasons for poor test performance and high computational resources and time consumption in locating the cause of errors are as follows: When using operation replay, only user operations are replayed, without reproducing the spatial state of the test scene at the time of user operation. This leads to different results for the same test case in different executions, resulting in poor test performance; when testing based on the commonly used methods, subtle changes in the spatial position, orientation, parent-child hierarchy, and relative relationships of virtual objects in AR / VR scenes result in poor consistency of spatial relationships, leading to poor test performance; when a test fails, the lack of complete scene state records makes it difficult to reproduce the real environment at the time of the problem, making it difficult to locate the cause of errors, consuming more computational resources and time. Based on this, some embodiments of this disclosure provide a test scenario recovery method, including: First, in response to detecting a test case recording operation and the interaction operation meeting preset scenario snapshot generation conditions, generating scenario snapshot information corresponding to the test scenario based on the object space information and object interaction state information of the current test scenario. This allows the generation of a scenario snapshot of the current test scenario for recording the current test scenario. Then, the scenario snapshot information and the interaction operation are associated and stored. This allows the recording of the scenario snapshot generated in the current test scenario and the corresponding interaction operation. Finally, in response to detecting a test case playback operation and the interaction operation to be executed meeting preset test recovery conditions, the test scenario is recovered based on the scenario snapshot information corresponding to the interaction operation to be executed. This allows the test scenario to be recovered for error localization. Because it does not merely play back user operations, but records the test scenario by generating a scenario snapshot of the current test scenario, the spatial state of the test scenario during user operations can be reproduced, thereby improving the testing effect. Furthermore, because the scene snapshots are generated based on object spatial information and object interaction state information, the consistency of spatial relationships in AR / VR scenes can be improved. This, in turn, enhances testing effectiveness. Moreover, since the test environment can be reproduced through the generated scene snapshot information, and this information can accurately represent the real-world environment where problems occurred during testing, the difficulty of locating the cause of errors is reduced, thereby decreasing the computational resources consumed in error localization and shortening the testing time.

[0064] Further reference Figure 3 This illustrates a flow 300 of another embodiment of the test scenario recovery method. Flow 300 of the test scenario recovery method includes the following steps: Step 301: In response to the detection of a test case recording operation and the interaction operation meeting the preset scenario snapshot generation conditions, generate scenario snapshot information for the corresponding test scenario based on the object space information and object interaction state information of the current test scenario.

[0065] Step 302: Perform associated storage processing on scene snapshot information and interactive operations.

[0066] In some embodiments, the specific implementation of steps 301-302 and the resulting technical effects can be found in [reference needed]. Figure 2 Steps 201-202 in the corresponding embodiments will not be repeated here.

[0067] Step 303: In response to the detection of a test case replay operation and the interaction operation to be executed meeting the preset test recovery conditions, obtain the scene snapshot information of the corresponding interaction operation to be executed from the associated database.

[0068] In some embodiments, in response to detecting a replay operation of a test case and the interaction operation to be executed meeting preset test recovery conditions, the execution entity can obtain scene snapshot information corresponding to the interaction operation to be executed from the associated database.

[0069] Step 304: Load the scene snapshot information.

[0070] In some embodiments, the execution entity can load the scene snapshot information. In practice, the execution entity can call an information loading interface to load the scene snapshot information. The information loading interface is an API interface used to load the scene snapshot information.

[0071] Optionally, the aforementioned execution entity can also call the information loading interface through the Unity engine to load and process the aforementioned scene snapshot information.

[0072] Step 305: Restore the test scene based on the loaded scene snapshot information.

[0073] In some embodiments, the aforementioned execution entity can restore the test scene based on the loaded scene snapshot information. In practice, the aforementioned execution entity can call the test scene restoration interface to restore the test scene based on the loaded scene snapshot information. The test scene restoration interface is an API interface used for restoring test scenes.

[0074] Alternatively, the aforementioned execution entity can also call the test scene restoration interface through the Unity engine to restore the test scene based on the loaded scene snapshot information.

[0075] from Figure 3 It can be seen from this that, with Figure 2 Compared to the description of some corresponding embodiments, Figure 3 The process 300 of the test scenario recovery method in some corresponding embodiments embodies the steps to restore the test scenario. Therefore, when replaying test cases, when a snapshot of the scenario with interactive operations to be executed is generated, the snapshot of the scenario with interactive operations to be executed can be directly loaded to restore the test scenario. This reduces the number of times the interactive operations to be executed need to be executed again, and the subsequent test steps in the replayed test cases can be executed in the restored test scenario. This restores the real interaction conditions, improves the effectiveness of the test scenario, shortens the test time, and reduces the computational resources consumed by re-executing interactive operations.

[0076] The following is for reference. Figure 4 It shows a schematic diagram of the structure of an electronic device 400 (e.g., a head-mounted display device or a terminal device) suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0077] like Figure 4 As shown, the electronic device 400 may include a processing unit 401 (e.g., a central processing unit, a graphics processor) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the electronic device 400. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0078] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 4 Each box shown can represent a device or multiple devices as needed.

[0079] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined above in the methods of some embodiments of this disclosure.

[0080] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: a communication connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0081] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0082] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the head-mounted display device to: in response to detecting a test case recording operation and the interaction operation satisfying preset scene snapshot generation conditions, generate scene snapshot information corresponding to the test scene based on the object space information and object interaction state information of the current test scene; perform associated storage processing on the aforementioned scene snapshot information and the aforementioned interaction operation; and in response to detecting a test case playback operation and the interaction operation to be executed satisfying preset test recovery conditions, perform test scene recovery processing based on the scene snapshot information corresponding to the interaction operation to be executed.

[0083] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0084] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0085] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0086] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A test scene recovery method, applied to a head-mounted display device, comprising: In response to the detection of a test case recording operation, and if the interaction operation meets the preset scenario snapshot generation conditions, the corresponding test scenario snapshot information is generated based on the object space information and object interaction state information of the current test scenario. The scene snapshot information and the interactive operation are associated and stored together. In response to the detection of a test case replay operation, and if the interaction operation to be executed meets the preset test recovery conditions, the test scenario is restored according to the scene snapshot information corresponding to the interaction operation to be executed.

2. The method according to claim 1, wherein, The method further includes: Collect object space information for the current test scenario; Collect interactive data information for the current test scenario.

3. The method according to claim 1, wherein, The associated storage processing of the scene snapshot information and the interactive operation includes: The scene snapshot information and the interactive operation are stored in the associated database.

4. The method according to claim 3, wherein, The step of restoring the test scene based on the scene snapshot information corresponding to the interactive operation to be executed includes: Retrieve scene snapshot information corresponding to the interactive operation to be executed from the associated database; The scene snapshot information is loaded and processed; The test scene is restored based on the loaded scene snapshot information.

5. The method according to claim 4, wherein, The step of generating scene snapshot information for the corresponding test scenario based on the object space information and object interaction state information of the current test scenario includes: The object space information and the object interaction state information are converted to obtain formatted object space information and object interaction state information. Based on the formatted object space information and object interaction state information, determine the scene snapshot information for the corresponding test scenario.

6. A head-mounted display device, comprising: One or more processors; Storage device for storing one or more programs; Optical mechanisms and optical components used to create images in front of the user's eyes; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

7. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images