Consistency tracing method, system and device for simulation test and real machine test, and storage medium

Abstract

This application discloses a consistency traceability method, system, device, and storage medium for simulation testing and real-machine testing, belonging to the field of navigation technology. It can solve the problem of inconsistent analysis between simulation testing and real-machine testing. The method includes: generating test cases corresponding to the target scenario based on scenario description information and test case specification templates; synchronously distributing the test cases to both the simulation end and the real-machine end; when either the simulation end or the real-machine end issues a decision request, generating structured intermediate instructions based on the test cases and the decision request to drive the simulation end or the real-machine end to execute the test cases in the corresponding execution window; associating and recording the first relevant data generated by the simulation end when executing the test cases with the second relevant data generated by the real-machine end when executing the test cases in a traceable execution record; and outputting a consistency evaluation report of the simulation end and the real-machine end in the target scenario based on the traceable execution record.

Description

Technical Field

[0001] This application belongs to the field of navigation technology, specifically relating to a method, system, device, and storage medium for consistency traceability between simulation testing and actual testing. Background Technology

[0002] In the process of researching navigation technology, it is generally necessary to first conduct simulation tests on the simulation end. If the simulation is error-free, then tests are conducted on the actual device to identify any problems that may exist during actual operation.

[0003] In existing technologies, the test results of the same navigation task on the simulation end and the actual device end generally differ significantly, making it impossible to directly align and compare them, and also impossible to verify the consistency of the test process. Summary of the Invention

[0004] The purpose of this application is to provide a method, system, device, and storage medium for consistency traceability between simulation testing and actual testing, which can solve the problem that simulation testing and actual testing cannot be compared and verified.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a consistency traceability method between simulation testing and actual testing, the method comprising: Generate test cases corresponding to the target scenario based on the scenario description information and the test case specification template; The test cases are simultaneously distributed to both the simulation and the actual machine. When a decision request is issued by either the simulation terminal or the physical terminal, a structured intermediate instruction is generated based on the test case and the decision request to drive the simulation terminal or the physical terminal to execute the test case in the corresponding execution window. The first relevant data generated by the simulation terminal when executing the test case and the second relevant data generated by the real machine terminal when executing the test case are associated and recorded in the traceable execution record; Based on the traceable execution records, a consistency evaluation report of the simulation terminal and the actual terminal in the target scenario is output.

[0006] Optionally, generating test cases based on scenario description information and test case specification templates includes: Based on the scene description information, read the scene description file and related dependency files corresponding to the scene description information, and perform normalization processing on the scene description file and related dependency files to obtain the normalized scene description file; Generate a summary and / or hash value corresponding to the normalized scene description file to obtain the scene identity identifier; Based on the normalized scene description file, determine the traversable representation corresponding to the target scene and generate the corresponding traversable hash value; Based on the standardized scene description file, the coordinate system of the target scene is fixed, the units in the target scene are unified, the coordinate transformation relationship of the target scene is recorded, and the header of the test case is generated. The test case parameters are determined according to the test case specification template. The test case parameters include at least the starting pose and tolerance of the target under test in the target scenario, the target definition and success criteria, the instruction text, the passage constraints and termination conditions, the indicator statistical standards, the alignment anchor point trigger criteria and the recording specifications. The test cases are generated based on the scenario identity identifier, the passable hash value, the test case header, and the test case parameters.

[0007] Optionally, based on the test cases and the decision request, structured intermediate instructions are generated, including: Read the observation data contained in the decision request; The structured intermediate instructions are generated based on the test cases and the observation data.

[0008] Optionally, the structured intermediate instructions are generated based on the test cases and the observation data, including: Based on the test cases, determine the specified start and end positions in the target scenario; Based on the observation data, determine the events in the target scene, the position and attitude of the target to be measured; Based on the starting point position, the ending point position, the events in the target scene, the position and attitude of the target to be tested, and a preset navigation strategy, corresponding window parameters and action parameters are generated to obtain the structured intermediate instructions.

[0009] Optionally, the step of associating and recording the first relevant data generated by the simulation terminal when executing the test case with the second relevant data generated by the real machine terminal when executing the test case in the traceable execution record includes: In the event of generating any instruction, the instruction identifier, instruction sequence number, execution window number, timestamp and / or step sequence, and observation summary of the instruction are recorded in the traceable execution record; The window start boundary, window end boundary, and execution summary of the simulation terminal and the actual terminal when executing the test case are respectively recorded in the traceable execution record; The corresponding event that triggers the decision request will be recorded in the traceable execution record; Related data from the same round of decision-making will be linked and recorded.

[0010] Optionally, the step of outputting a consistency evaluation report of the simulation terminal and the actual terminal in the target scenario based on the traceable execution record includes: Read the traceable execution record, and read the first relevant data and the second relevant data; Align the first relevant data with the second relevant data to determine the difference location conclusion between the first relevant data and the second relevant data; Based on the findings of the difference localization, the consistency assessment report is obtained.

[0011] Optionally, the step of aligning the first relevant data and the second relevant data to determine the difference location conclusion between the first relevant data and the second relevant data includes: Based on the alignment anchor points, window information, and step sequence recorded in the first and second related data, the differences between the simulation end and the actual machine end in the corresponding window are determined; The difference localization conclusion is obtained based on the execution summary of the window where the difference point is located.

[0012] Secondly, embodiments of this application provide a consistency traceability system between simulation testing and actual machine testing, the system comprising: The consistency test case generation module is used to generate test cases corresponding to the target scenario based on the scenario description information and the test case specification template, and to distribute the test cases to the inference server module and the client execution module. The inference server module is used to receive the decision request sent by the client execution module, generate the corresponding structured intermediate instructions based on the test cases and the observation data in the decision request, and combine them with the preset navigation strategy, and send the structured intermediate instructions to the client execution module. The client execution module is used to control the simulation terminal and the actual terminal to execute the test cases according to the structured intermediate instructions, and send the first relevant data generated by the actual terminal and the second relevant data generated by the simulation terminal to the traceable record and regression comparison module. The data transmission and recording association module is used to transmit observation data and structured intermediate instructions between the inference server module and the client execution module, and write association fields such as instruction identifier and output sequence number into the traceable execution record; The traceable record and regression comparison module is used to save the first related data and the second related data, align the first related data and the second related data according to the alignment anchor point, window information and step sequence, and output a consistency assessment report.

[0013] Thirdly, embodiments of this application provide a consistency traceability device between simulation testing and actual testing, the device comprising: The test case generation module is used to generate test cases corresponding to the target scenario based on the scenario description information and the test case specification template. The test case synchronization module is used to synchronously distribute the test cases to the simulation terminal and the actual machine terminal; The instruction generation module is used to generate structured intermediate instructions based on the test cases and the decision request when either the simulation end or the actual machine end issues a decision request, so as to drive the simulation end or the actual machine end to execute the test cases in the corresponding execution window. The data recording module is used to associate and record the first relevant data generated by the simulation terminal when executing the test case with the second relevant data generated by the real machine terminal when executing the test case into a traceable execution record; The report generation module is used to output a consistency assessment report of the simulation terminal and the actual terminal in the target scenario based on the traceable execution record.

[0014] Optionally, the test case generation module includes: The first data generation submodule is used to read the scene description file and related dependency files corresponding to the scene description information according to the scene description information, and to perform normalization processing on the scene description file and the related dependency files to obtain a normalized scene description file. The second data generation submodule is used to generate a summary and / or hash value corresponding to the normalized scene description file to obtain the scene identity identifier; The third data generation submodule is used to determine the feasible representation corresponding to the target scene based on the normalized scene description file, and generate the corresponding feasible hash value. The fourth data generation submodule is used to solidify the coordinate system of the target scene, unify the units in the target scene, record the coordinate transformation relationship of the target scene, and generate the header of the test case based on the normalized scene description file. The fifth data generation submodule is used to determine the test case parameters of the test case according to the test case specification template. The test case parameters include at least the starting pose and tolerance of the target to be tested in the target scenario, the target definition and success criteria, the instruction text, the passage constraints and termination conditions, the indicator statistical standards, the alignment anchor point trigger criteria and the recording specifications. The test case generation submodule is used to generate the test cases based on the scenario identity identifier, the passable hash value, the test case header, and the test case parameters.

[0015] Optionally, the instruction generation module includes: The observation data reading submodule is used to read the observation data contained in the decision request; The instruction generation submodule is used to generate the structured intermediate instructions based on the test cases and the observation data.

[0016] Optionally, the instruction generation submodule includes: The target determination submodule is used to determine the specified start and end positions in the target scenario based on the test cases. The location determination submodule is used to determine the location and attitude of events and the target to be measured in the target scene based on the observation data. The instruction determination submodule is used to generate corresponding window parameters and action parameters based on the starting position, the ending position, the events in the target scene, the position and attitude of the target to be tested, and a preset navigation strategy, so as to obtain the structured intermediate instruction.

[0017] Optionally, the data recording module includes: The first recording submodule is used to record the instruction identifier, instruction sequence number, execution window number, timestamp and / or step sequence, and observation summary of any instruction generated into the traceable execution record; The second recording submodule is used to record the window start boundary, window end boundary, and execution summary of the simulation terminal and the actual terminal when executing the test case to the traceable execution record; The third recording submodule is used to record the corresponding event that triggers the decision request to the traceable execution record; The associated record submodule is used to associate and record related data corresponding to the same round of decisions.

[0018] Optionally, the report generation module includes: The data reading submodule is used to read the traceable execution record and to read the first related data and the second related data. The alignment processing submodule is used to align the first related data and the second related data, and determine the difference location conclusion between the first related data and the second related data; The report generation submodule is used to obtain the consistency assessment report based on the difference location conclusion.

[0019] The step of aligning the first relevant data and the second relevant data to determine the difference location conclusions between the first relevant data and the second relevant data includes: The difference point determination submodule is used to determine the difference points between the simulation end and the actual machine end in the corresponding window based on the alignment anchor points, window information and step sequence recorded in the first relevant data and the second relevant data; The difference localization conclusion determination submodule is used to obtain the difference localization conclusion based on the execution summary of the window where the difference point is located.

[0020] Fourthly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0021] Fifthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0022] In a sixth aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0023] The consistency traceability method for simulation testing and real-machine testing provided in this application generates test cases corresponding to the target scenario based on scenario description information and test case specification templates; the test cases are synchronously distributed to the simulation end and the real-machine end; when either the simulation end or the real-machine end issues a decision request, structured intermediate instructions are generated based on the test cases and the decision request to drive the simulation end or the real-machine end to execute the test cases in the corresponding execution window; the first relevant data generated by the simulation end when executing the test cases and the second relevant data generated by the real-machine end when executing the test cases are associated and recorded in a traceable execution record; based on the traceable execution record, a consistency evaluation report of the simulation end and the real-machine end in the target scenario is output.

[0024] In this method, unified test cases are generated based on the same test scenario and synchronously distributed to both the simulation and the actual device. When the simulation and the actual device send decision requests, corresponding structured intermediate instructions are generated to drive the execution of the test cases to complete the target test task. At the same time, relevant data from the execution of test cases by the simulation and the actual device are recorded in a traceable execution record. The traceable execution record is analyzed to output a corresponding consistency evaluation report. Based on the same test scenario and the same test cases, the simulation and the actual device are synchronously driven to perform tests, and then the relevant data in the test is evaluated for consistency. This completes the alignment and consistency evaluation of the test tasks of the simulation and the actual device, improving the development efficiency of the navigation system. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the structure of a cross-platform execution and consistency regression testing system proposed in an embodiment of this application; Figure 2 This is a flowchart of a consistency traceability method between simulation testing and actual testing proposed in an embodiment of this application; Figure 3 This is a schematic diagram of a test case generation process proposed in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the time-domain alignment of the structured intermediate instructions and execution window mechanism proposed in an embodiment of this application; Figure 5 This is a schematic diagram of a consistency traceability device for simulation testing and actual testing proposed in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device proposed in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0028] This application's embodiments are based on cross-platform execution and consistency regression testing system execution, referencing... Figure 1 , Figure 1 This is a schematic diagram of the cross-platform execution and consistency regression testing system proposed in one embodiment of this application, as shown below. Figure 1 As shown, the system includes a consistency test case generation module, an inference decision-making module (server-side), a client execution module (including simulation and physical machine ends), a data transmission and recording association module, and a traceable record and regression comparison module. The consistency test case generation module outputs test case packages and distributes them to the simulation and physical machine ends. The client collects observations and transmits observation data and structured intermediate instructions between the decision-making and execution ends through the data transmission and recording association module. The decision-making end outputs structured intermediate instructions and execution window fields. The client executes these instructions frequently according to the execution window and records relevant data to the traceable execution record. The traceable record and regression comparison module aligns the traceable execution records based on anchor points and window boundaries and outputs a consistency assessment and a difference summary.

[0029] After generating test cases, the consistency test case generation module sends them to the inference decision-making module and the client execution module. Either the simulation or the actual machine in the client execution module can send a decision request to the server. Based on the observation data included in the decision request, the server combines the corresponding navigation strategy with the starting and target positions specified in the test cases to generate corresponding structured intermediate instructions. Data transmission and recording are completed between the client and server through the data transmission and recording association module. The data generated by the client when executing the test case is recorded in the traceable record and regression comparison module. Finally, the traceable record and regression comparison module outputs a consistency evaluation report. The detailed process is described later.

[0030] The consistency traceability method between simulation testing and actual testing provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0031] refer to Figure 2 , Figure 2 This is a flowchart of a consistency traceability method between simulation testing and actual testing proposed in an embodiment of this application, as shown below. Figure 2 As shown, the method specifically includes the following steps: S11: Generate test cases corresponding to the target scenario based on the scenario description information and the test case specification template.

[0032] In this embodiment, the scene description information includes a scene description file (USD file, Universal SceneDescription), related dependency files, and a case specification schema, which is a pre-established set of field structures and semantic constraints used to define the elements that a case instance must include, the statistical standards for metrics, the expression method of alignment anchors, and the TraceLog (traceable execution record) recording specifications, and includes a template identifier and version information. The traceable execution record records a combination of time-series records, event stream records, and replay indexes required for traceability, used for review, alignment comparison, and regression attribution of differences.

[0033] In this embodiment, the consistency test case generation module standardizes the scene description file and related dependency files in the scene description information, thereby generating corresponding digests or hash values. Then, based on the scene description information, it generates a corresponding feasible representation and a feasible representation hash value. Next, based on the spatial structure in the scene description information and the relevant data of the target under test, it solidifies the coordinate system, units, coordinate system mapping relationships, etc., and generates the test case header. Under the constraints of the test case specification template, it generates the starting point pose and tolerance, target definition and success criteria, instruction text and template information, access constraints and termination conditions, indicator statistical standards, alignment anchor point triggering criteria, and recording specifications. Finally, it validates the generated field structure and the constraints, and outputs the test cases.

[0034] For example, the test scenario is an indoor navigation scenario, and the target under test is a robot, that is, navigating the robot in an indoor environment. The test scenario can also be a road scenario, and the target under test is a car, that is, navigating the car in a road scenario.

[0035] S12: Simultaneously distribute the test cases to both the simulation end and the actual machine end.

[0036] In this embodiment, after generating test cases, the test cases are distributed to the system's decision-making end and client. The decision-making end is used to make navigation decisions, and the client includes a simulation end and a real machine end. The simulation end is the target being tested in a simulation environment, and the real machine end is the target being tested in a real scenario.

[0037] S14: When a decision request is issued by either the simulation terminal or the actual machine terminal, a structured intermediate instruction is generated based on the test case and the decision request to drive the simulation terminal or the actual machine terminal to execute the test case in the corresponding execution window.

[0038] In this embodiment, the structured intermediate instructions are instructions generated by the decision-making end based on the received decision request during test execution. The corresponding execution window is an execution window generated by the decision-making end based on the decision request during test case execution. The execution window specifies a preset number of execution steps, and each test case can be divided into multiple execution windows for execution. The window parameters can be continuously updated based on observation data. The decision request includes at least the observation data of the current environment collected by the simulation end or the test end through the image acquisition device. Based on this data, the position and attitude of the target under test (simulation end or real machine end) in the scene can be determined.

[0039] In this embodiment, when the simulation end or the actual machine end executes the test cases and moves in the target scene according to the target task in the test cases, it sends a decision request to the server. When a preset event occurs during the operation of the simulation end or the decision end, the execution window will be closed immediately and a decision request will be sent to the decision end. The decision request carries the corresponding observation data. The inference service module generates structured intermediate instructions according to the preset navigation strategy, such as the VLN (Vision-Language Navigation) strategy. The structured intermediate instructions are sent to the request initiator through the data transmission and recording association module, driving the request initiator to execute the test cases in the corresponding execution window.

[0040] S15: Associate the first relevant data generated by the simulation terminal when executing the test case with the second relevant data generated by the real machine terminal when executing the test case and record it in the traceable execution record.

[0041] In this embodiment, the first and second related data include at least the window start time / step count, window end time / step count, cumulative commands executed within the window, execution summary, window end reason, and window number. It also includes related fields such as instruction identifier and instruction sequence number of intermediate instructions. If a reply is enabled, it also includes the execution reply standard statement and the recovery result.

[0042] In this embodiment, the data transmission and recording association module associates and records the first relevant data generated by the simulation end when executing test cases and the second relevant data generated by the real machine end when executing test cases into the traceable execution record, which facilitates subsequent alignment, unification and consistency evaluation.

[0043] S16: Based on the traceable execution record, output a consistency evaluation report of the simulation terminal and the actual terminal in the target scenario.

[0044] In this embodiment, after the test cases are executed, the traceable execution records are read, the first relevant data and the second relevant data are aligned and unified, the differences are identified, the differences are analyzed, and then a consistency evaluation report of the simulation end and the actual machine end in the target scenario is output.

[0045] In this embodiment, a test case is generated for the same test scenario and the target under test. The same instructions drive the simulation end and the actual machine end to execute the test case. The relevant data when the simulation end and the actual machine end execute the test case are recorded in association, and then a consistency evaluation report is generated. This enables the analysis of differences between simulation testing and actual machine testing, thereby improving the development efficiency of the navigation system.

[0046] In another embodiment of this application, generating test cases based on scenario description information and test case specification templates includes: S21: Based on the scene description information, read the scene description file and related dependency files corresponding to the scene description information, and perform normalization processing on the scene description file and related dependency files to obtain a normalized scene description file.

[0047] In this embodiment, the standardized scenario description file includes a main file and a dependency list. The main file defines the test scenario and test information, and the dependency list includes a list of files required for the test.

[0048] In this embodiment, the system obtains scene description information, reads the scene description file and related dependency files corresponding to the scene description information, and performs normalization processing on the scene description file and related dependency files to obtain a normalized scene description file. The normalization processing method includes at least unifying resource path representation, removing non-deterministic metadata that is irrelevant to the test, and sorting the dependency asset list according to a preset order.

[0049] S22: Generate a summary and / or hash value corresponding to the normalized scene description file to obtain the scene identity identifier.

[0050] In this embodiment, the summary is a field that provides a brief description of the file content, and the hash value is a field obtained after performing a hash operation on the file. Using it as a scenario identifier can be used for data integrity assurance, version control, and system traceability.

[0051] In this embodiment, after obtaining the standardized scene description file, a corresponding summary or hash value is generated to obtain the scene identity identifier.

[0052] S23: Based on the normalized scene description file, determine the passable representation corresponding to the target scene and generate the corresponding passable hash value.

[0053] In this embodiment, a traversable representation is a semantic description of which areas in the scene can be moved or navigated. A traversable hash value is a unique identifier for a traversable representation in the scene, used to ensure consistency and traceability across different systems or versions.

[0054] In this embodiment, a topology is generated by analyzing the geometry (such as walls, doors, and passages) in the USD scene. Key points (such as passage entrances, intersections, and leading edges) are extracted from the scene and used as nodes in the navigation path. The scene is divided into grids or regions, and each region is marked as passable or impassable. This yields a passable representation. A hash operation is then performed on the passable representation to obtain a passable hash value.

[0055] S24: Based on the standardized scene description file, solidify the coordinate system of the target scene, unify the units in the target scene, record the coordinate transformation relationship of the target scene, and generate the header of the test case.

[0056] In this embodiment, based on the description information of the geometry in the scene and the path information in the scene in the normalized scene description file, the coordinate system of the target scene is fixed, the units in the target scene are unified, and the coordinate transformation relationship in the target scene is recorded. This information is used as the header of the test case.

[0057] S25: Determine the test case parameters according to the test case specification template. The test case parameters include at least the starting pose and tolerance of the target to be tested in the target scenario, the target definition and success criteria, the instruction text, the passage constraints and termination conditions, the indicator statistical standards, the alignment anchor point triggering criteria and the recording specifications.

[0058] In this embodiment, test case parameters are generated based on the starting position and tolerance (the starting position and allowable error range of the target under test), target definition and success criteria (the execution objective of the test case and the basis for determining the success of the test case execution), instruction text (the macro-level instructions on how the target under test moves to the target point), passage constraints and termination conditions (the constraints followed during passage and the termination conditions during passage), indicator statistical standards (which indicators need to be statistically analyzed and how to statistically analyze them), alignment anchor point triggering criteria and recording specifications (key events / states that can be detected and recorded during execution, used for cross-end alignment and segment comparison, which at least includes anchor point identifier, triggering criteria, tolerance range, and debouncing rules) specified in the standardized test case template.

[0059] S26: Generate the test case based on the scenario identity identifier, the passable hash value, the packet header of the test case, and the test case parameters.

[0060] In this embodiment, the scene identity identifier, passable hash value, test case header, and test case parameters are filled into the test case template according to preset rules to obtain the test case.

[0061] refer to Figure 3 , Figure 3 This is a schematic diagram of a test case generation process proposed in an embodiment of this application, as shown below. Figure 3 As shown, the system collects and standardizes the USD master file and its dependent assets (scene description files), generates a scene identity identifier (scene_hash), and generates a summary or hash value for the standardized master file and dependency list. It reads or generates a passable representation and generates a passability hash value (passability_hash). The coordinate system, units, and calibration / coordinate mapping summary are then solidified and written into the test case package header to achieve version binding and cross-platform interpretation consistency. Under the constraints of the test case specification template, the system generates the starting point pose and tolerance, target definition and success criteria, instruction text and template information, passability constraints and termination conditions, indicator statistical standards, alignment anchor point triggering criteria and recording specifications. Alignment anchor points can be generated from USD semantic annotations, geometric structure detection results, or topological key points in the passable representation. Anchor point types include at least plane traversal, entry into a region, and arrival at the forefront / fork in the road. Finally, a validator verifies the field structure and semantic constraints and outputs the test case package.

[0062] In this embodiment, generating unified test cases based on the scenario description file and test case template facilitates consistency testing and evaluation between the simulation and testing ends. By generating unified test cases, setting unified task elements and statistical standards, introducing alignment anchors and recording them in a traceable execution log, the same test case can be reproducibly executed, aligned, compared, and regressively verified on different platforms, versions, and operating conditions. Through the automated generation chain from USD to test case packages, any USD scenario is transformed into a regression test package containing task specifications, instructions, statistical standards, alignment anchors, and recording specifications. This significantly reduces manual configuration and version adaptation costs, and improves test case coverage and the automation and maintainability of the regression pipeline. The above methods achieve consistency testing between the simulation and testing ends, improving the verification efficiency of the visual navigation system and thus enhancing the system's development efficiency.

[0063] In another embodiment of this application, structured intermediate instructions are generated based on the test cases and the decision request, including: S31: Read the observation data contained in the decision request.

[0064] In this embodiment, when the server receives a decision request from the simulation end or the actual machine end, it reads the observation data contained in the decision request.

[0065] S32: Generate the structured intermediate instructions based on the test cases and the observation data.

[0066] In this embodiment, the task objectives of the test task are determined according to the test cases, such as the starting point, the ending point, and the approximate process of movement. The location and current posture of the robot on the simulation end or the robot on the test end can be determined based on the observation data. Then, combined with the preset navigation strategy, the corresponding structured intermediate instructions are generated.

[0067] In another embodiment of this application, the structured intermediate instructions are generated based on the test cases and the observation data, including: S41: Based on the test cases, determine the specified start and end positions in the target scenario.

[0068] In this embodiment, the server can determine the starting and ending positions of the target to be tested in the target scenario based on the test cases, and thus determine the approximate movement route.

[0069] S42: Based on the observation data, determine the events in the target scene, the position and attitude of the target to be measured.

[0070] In this embodiment, the server determines the events, the location, and the attitude of the target in the target scene based on the observation data uploaded by the client.

[0071] In this embodiment, when the simulation end and the actual machine end execute test cases, they need to move to the target location in the test cases. After receiving the decision request, the server does not send a request to the decision-making end by default during the execution window. When any preset condition is triggered, a decision request is sent to the decision-making end. The decision request includes the events in the target scene, the position and attitude of the target to be tested. The decision-making end reads the decision request, receives the observation data, and determines the events in the target scene, the position and attitude of the target to be tested contained therein.

[0072] S42: Based on the starting position, the ending position, the events in the target scene, the position and attitude of the target to be tested, and a preset navigation strategy, generate corresponding window parameters and action parameters to obtain the structured intermediate instructions.

[0073] In this embodiment, the server generates structured intermediate instructions based on the location of the target under test and the starting and ending positions of the task target. The structured intermediate instructions include window parameters and action parameters, so that the request initiator can perform the corresponding action in the corresponding execution window, ensuring that the target under test in the scene moves in the manner specified by the test case, thereby realizing the navigation of the target under test.

[0074] For example, the window parameter is time_to_go (execution time) / steps_to_go (execution steps).

[0075] For example, the action set may include, but is not limited to: MOVE, TURN, STOP, SIDESTEP, etc., and the parameter range (velocity / angular velocity / acceleration limit, etc.) may be defined in the use case package or system configuration.

[0076] In this embodiment, after receiving the instructions output by the decision-making end, the client further processes the instructions. This processing includes action type whitelist verification, parameter range verification and pruning, and window field verification (range of time_to_go / steps_to_go, minimum execution time, etc.). Only actions in the action type whitelist can be executed. The parameter range must conform to the behavior patterns of the target being tested; otherwise, the parameters are pruned to conform to a preset range. After processing the instructions, a window number is generated and recorded.

[0077] In this embodiment, the decision-making end outputs structured intermediate instructions `mid_cmd` and window parameters `time_to_go` or `steps_to_go` at a low frequency. The client performs action whitelist and parameter range verification on the intermediate instructions and can perform safe pruning. Then, it generates a window number `window_id` and records the window start boundary. Within the window, the client executes underlying control instructions at high frequency according to the control cycle `control_dt` and records the state and indicator increment within the window. When conditions such as abnormal event triggering, anchor point hit, sensor / positioning degradation, or stop flag are met, the client preempts the termination window and records the window end boundary and the end reason `end_reason`, and triggers the next decision call to obtain the structured intermediate instructions for the subsequent window, thereby establishing a cross-end reproducible time-domain alignment benchmark.

[0078] refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the time-domain alignment of the structured intermediate instructions and execution window mechanism proposed in an embodiment of this application, as shown below. Figure 4As shown, the decision-making end outputs structured intermediate instructions `mid_cmd` and window parameters `time_to_go` or `steps_to_go` at a low frequency. The client performs action whitelist and parameter range verification on the intermediate instructions and can perform safe pruning. Then, it generates a window number `window_id` and records the window's start boundary. Within the window, the client executes underlying control instructions at a high frequency according to the control cycle `control_dt` and records the window's state and indicator increments. When conditions such as abnormal event triggering, anchor point hit, sensor / positioning degradation, or stop flag are met, the client preempts the termination window and records the window's end boundary and the end reason `end_reason`, and triggers the next decision call to obtain the structured intermediate instructions for the subsequent window, thereby establishing a cross-end reproducible time-domain alignment benchmark.

[0079] In this embodiment, when the decision-making end outputs `time_to_go_s`, the client calculates `steps_to_go = ceil(time_to_go_s / control_dt)` according to the control cycle `control_dt`, or directly uses the `steps_to_go` output by the decision-making end. The client assigns a `window_id` to each window and records the window's start step, window's end step, and the cumulative executed commands and execution summary within the window, serving as a stable benchmark for cross-end alignment.

[0080] In another embodiment of this application, the step of associating and recording the first relevant data generated by the simulation terminal when executing the test case with the second relevant data generated by the real machine terminal when executing the test case in a traceable execution record includes: S51: When any instruction is generated, the instruction identifier, instruction sequence number, execution window number, timestamp and / or step sequence, and observation summary of the instruction are recorded in the traceable execution record.

[0081] In this embodiment, when any instruction is generated, the instruction identifier, instruction sequence number, corresponding execution window number, timestamp and / or step sequence, and observation summary of the instruction are recorded in the traceable execution record.

[0082] S52: Record the window start boundary, window end boundary, and execution summary of the simulation terminal and the actual terminal when executing the test case to the traceable execution record.

[0083] In this embodiment, when executing test cases on the simulation end and the actual machine end, the start boundary of the execution window, the end boundary of the window, and the execution summary are recorded in the traceable execution record.

[0084] S53: Record the corresponding event that triggered the decision request to the traceable execution record.

[0085] In this embodiment, when a preset event is triggered to send a decision request, the corresponding event is recorded in the traceable execution record.

[0086] S54: Link and record relevant data corresponding to the same round of decisions.

[0087] In this embodiment, in each round of decision-making, the instruction identifier, instruction sequence number, execution window number, timestamp and / or step sequence, observation summary, the window start boundary, window end boundary, execution summary and corresponding events are associated and recorded, which is beneficial for subsequent consistency analysis.

[0088] In this embodiment, the data transmission and record association module transmits observation data and structured intermediate instructions (implementation method is not limited) between the decision-making end (e.g., inference server or local inference component) and the execution end (simulation end or physical machine end), and establishes a traceable "record association" link for consistency regression testing. Specifically, the module associates the test case instance identifier test_case_id and scene version binding information (scene_hash, passability_hash, coord_mapping_digest) with the instruction identifier / output sequence number cmd_seq (or server_output_seq) of each decision output, the corresponding execution window number window_id, and the execution end timestamp / step sequence (client_ts / client_step_id) and observation summary obs_digest. Related fields are passed through and written to TraceLog, so that "observation input - decision output - window execution - event / recovery - indicator statistics" can be played back in the log. At the same time, it is required to record the start / end boundary of the window and the reason for the end, the difference between the original intermediate instructions and the instructions after the execution side is clipped, and write the playback index (key frame / video / trajectory segment location) when the event is triggered, so as to support cross-end alignment comparison and difference location.

[0089] In this embodiment, the abnormal event criteria can be: ANOMALY_STUCK (Stuck): The displacement increment is less than the threshold ε within a duration T, and the cumulative control commands within the window are non-zero. ANOMALY_COLLISION (Collision Anomaly): The number of collision / contact events exceeds the threshold within a unit of time, or continuous collisions exceed the threshold. ANOMALY_OFFROUTE (Deviation): The distance from the current position to the allowed passage corridor / planned route exceeds the threshold. ANCHOR_FAIL_DOOR (Doorway Alignment Failure): The doorway anchor point criterion is not met, and multiple attempts fail. Events should record at least the trigger threshold, time, window, observation summary, pose / velocity, structured intermediate commands, and lower-level execution summary, among other system status information.

[0090] In this embodiment, to improve reproducibility and comparability, the recovery strategy (the strategy to restore the target under test to normal operation when an abnormal event is triggered) is fixed in the form of a "primitive sequence" (such as BACKOFF, INPLACE_TURN, SIDESTEP, STOP_SCAN, RELOCALIZE), and the following are recorded: recovery number, action sequence, duration, success or failure, and whether the next intermediate instruction acquisition is triggered after recovery. The recovery logic can be optionally enabled, but once enabled, a unified recording standard must be used.

[0091] In this embodiment, the traceable execution record includes at least time-series records, event stream records, and playback material indexes. Event stream records include at least the event type, trigger criteria and thresholds, status information, recovery sequence, and result. The playback material index includes at least the path to the video / keyframe / track / log file and its corresponding timestamp or step mapping. The time series can be stored using JSONL (one record per line) or binary + index method to balance efficiency and readability.

[0092] In another embodiment of this application, the step of outputting a consistency evaluation report of the simulation terminal and the actual terminal in the target scenario based on the traceable execution record includes: S61: Read the traceable execution record, and read the first related data and the second related data.

[0093] In this embodiment, after the test is completed, the traceable execution record is read through the traceable record and regression comparison module to obtain the first relevant data of the simulated robot and the second relevant data of the actual robot.

[0094] S62: Align the first relevant data and the second relevant data to determine the difference location conclusion between the first relevant data and the second relevant data.

[0095] In this embodiment, the difference localization conclusion is obtained by analyzing and localizing the differences between each window and the differences at each anchor point during the execution of the test cases based on the first relevant data and the second relevant data.

[0096] In this embodiment, the differences between each window are analyzed and located sequentially using anchor points, windows, and step sequences. The difference between the first and second relevant data points is obtained, and the observation summary at the difference point is analyzed to arrive at the difference location conclusion.

[0097] S63: Based on the difference location conclusion, the consistency assessment report is obtained.

[0098] In this embodiment, the differences between the simulation end and the actual machine end when executing test cases are determined based on the difference location conclusion. The overall situation of the simulation end and the actual machine end executing test cases and the differences are written into the report to obtain a consistency evaluation report.

[0099] In another embodiment of this application, the step of aligning the first related data and the second related data to determine the difference location conclusion between the first related data and the second related data includes: S71: Based on the alignment anchor points, window information, and step sequence recorded in the first and second related data, determine the differences between the simulation end and the actual machine end in the corresponding window.

[0100] In this embodiment, the differences include key indicator differences, trajectory differences, trigger event differences, whether a recovery event occurred, and the recovery result. Both the first and second related data record the alignment anchor point, window information, and the command execution sequence (step number).

[0101] In this embodiment, the traceable record and regression comparison module prioritizes using alignment anchors to align the time in the first related data and the second related data. It can also align the time of the first related data according to the window boundary, window number, etc. of each execution window, and can also align the time of the first related data and the second related data according to the step sequence of instruction execution.

[0102] In this embodiment, the traceable record and regression comparison module determines the differences between the first and second related data in the aligned data.

[0103] S72: Based on the execution summary of the window where the difference point is located, the difference localization conclusion is obtained.

[0104] In this embodiment, after determining the difference points between the first relevant data and the second relevant data, the specific difference content at each difference point is determined based on the execution summary of the window where the difference point is located, thereby obtaining the difference location conclusion.

[0105] In this embodiment, segment alignment is preferentially performed using an alignment anchor point sequence. When there are insufficient anchor points, window boundary (window_id) alignment is used. When there are still insufficient anchor points, step sequence or timestamp alignment is used. Difference summary output: For each segment, the following are output: anchor point / window interval, key indicator difference, trajectory difference overview, trigger event difference, whether recovery occurred and whether recovery was successful, thereby locating "from which window did the deviation begin, what happened when the deviation occurred, and whether secondary differences were caused by recovery".

[0106] By using structured intermediate instructions and window execution mechanisms, the low-frequency inference of the upper layer and the high-frequency control of the lower layer are unified to a recordable window boundary (time_to_go / steps_to_go), reducing the repeated acquisition of intermediate instructions and reducing the impact of nondeterministic external disturbances on the closing cycle timing and alignment benchmark. At the same time, it supports window preemption triggered by abnormalities and the acquisition of subsequent window intermediate instructions, so that the system can maintain consistent timing, interpretable behavior, and comparable alignment under disturbance conditions.

[0107] In another embodiment of this application, the target scene is set as a room and a doorway, and the target to be detected is a robot.

[0108] The scenario and task are as follows: In the USD scenario, room_A and room_B are connected by a doorway_02 (the doorway is approximately 0.9m wide, and the door frame is semantically labeled as door_02, which can be projected as door_plane_02 in navmesh). The test case specifies that the target under test (robot) starts from room_A, with the instruction "proceed along the corridor, pass through door_02, enter room_B, and stop near sofa_area_01". The alignment anchor points include door_02 (pass_through) and goal_region (enter_region).

[0109] In one regression test, the actual device exhibited slippage / slight bumping against the door frame in front of the doorway entrance, resulting in it "appearing to move, but its pose was almost stationary," accompanied by continuous contact events. The system triggered events according to the following quantization rules: Stuck trigger: Within window 9, for a duration of T = 1.2s (e.g., 6 control cycles, control_dt = 0.2s), the following conditions are met: Position displacement increment Δpos < ε = 0.05m Furthermore, the average execution speed of lower-level instructions within the window, |v_cmd|, is greater than 0.15 m / s (non-zero drive). → Trigger a freeze and record the trigger threshold and statistics window.

[0110] Continuous collision trigger: Within the same window, if the cumulative number of collision / contact callbacks is >= 3 times / second or the continuous contact lasts for >0.6 seconds, continuous collisions are triggered, forming a "compound abnormal chain" with the jamming.

[0111] Event log content (Example of a TraceLog event stream segment) When triggered, the client writes the following minimum chain of evidence into the event stream (illustrated): Commands are configured: 38 Event type: "Stuck" Execution sequence: 1980 Window number: 9 Threshold parameters: { "Position change (meters)": 0.04, "Duration (seconds)": 1.2, "Minimum command speed (meters / second)": 0.15} The execution summary must contain at least: mid_cmd: Server intermediate commands (MOVE / velocity / angular velocity / window field) cmd_exec: The actual number of executed instructions and the cumulative execution time after trimming. pose / vel: Pose and velocity estimation at the trigger moment obs_digest: Keyframe index / image digest hash (used for playback localization) anchor_state: door_02 Whether the anchor point has been hit (not hit here).

[0112] If a collision exception is subsequently triggered, an additional event will be added: Event Type: "Chain Collision" Conflict count within the time window: N Contact duration: 0.8 Related anchor point: "door_02".

[0113] Execution and recording of standard restoring primitives After the system enables the recovery logic, a unified recovery primitive sequence is used for "stuck / collision in front of the doorway" (example): Recovery behavior: Reverse - Steering - Lateral movement Backward: Move backward 0.25 meters (or for 1.0 second) Turn on the spot: Turn 15° to the left (avoiding the door frame). Lateral movement: 0.15 meters (optional, enabled only when chassis support is available) Stop scanning: Pause for 0.5 seconds, acquire keyframes, and trigger the next round of intermediate instruction acquisition. Trace log records: Recovery behavior ID: BACKOFF_TURN_SIDESTEP Execution sequence: ["Back", "Turn in place", "Side shift", "Stop scanning"] Total time (seconds): 2.6 Execution result: Success (or failure) Execution result details: { "Seize the window of opportunity": true, "Next decision has been triggered": true, New window number: 10 } How can cross-end alignment and regression comparisons "locate where the differences begin and why they occur"?

[0114] The regression comparison module outputs a summary of differences based on "anchor point priority, followed by window priority": Simulation client (version A): No abnormal events in window_9; anchor point door_02 hit at step=2010; trajectory smoothly passes through the doorway.

[0115] On the actual device (version B): window_9 triggered a lag + chain collision; the strategy of reversing - turning - lateral movement was executed to recover.

[0116] preempt_window: Preempts the time window / schedule window, and then hits door_02 in window_11; ultimately success=true, but path_length_m and time_s increase.

[0117] Differential localization conclusions (example output): Starting point of difference: window_9 (before anchor point door_02) Direct cause: Continuous contact leading to abnormal collision + failure of displacement to increase leading to abnormal jamming. Secondary impact: Restore preemption → Add window → Door_02 anchor hit delay Chain of evidence: Event stream (threshold and execution summary) + playback index (keyframe / video) + window boundary (window_9 end_reason=PREEMPT_BY_ANOMALY).

[0118] This scenario is just an example; the test scenario could also be a car navigation system or similar scenario.

[0119] In this embodiment, quantifiable anomaly criteria and standard recovery primitives are added for various events. A unified event format is used to record triggering conditions, recovery sequences, and results, thereby reducing the probability of failures such as accidental entry and lag. It also supports event-based localization and review in regression comparisons, identifying "when the difference started, what triggered it, and how it was recovered." The system supports review audits and difference localization to windows / anchors / events and playback segments. Furthermore, by recording the differences between original intermediate instructions and execution-side trimmed instructions, it supports distinguishing between differences in decision outputs and differences caused by compliance restrictions.

[0120] It should be noted that the consistency traceability method between simulation testing and actual testing provided in this application embodiment is illustrated by taking the consistency traceability device between simulation testing and actual testing as an example to illustrate the consistency traceability method between simulation testing and actual testing provided in this application embodiment.

[0121] refer to Figure 5 , Figure 5 This is a schematic diagram of a consistency traceability device 500 for simulation testing and actual machine testing according to an embodiment of this application, as shown below. Figure 5 As shown, the device includes: The test case generation module 501 is used to generate test cases corresponding to the target scenario based on the scenario description information and the test case specification template. The test case synchronization module 502 is used to synchronously distribute the test cases to the simulation terminal and the actual machine terminal; The instruction generation module 503 is used to generate structured intermediate instructions based on the test cases and the decision request when either the simulation end or the actual machine end issues a decision request, so as to drive the simulation end or the actual machine end to execute the test cases in the corresponding execution window. The data recording module 504 is used to associate and record the first relevant data generated by the simulation terminal when executing the test case with the second relevant data generated by the real machine terminal when executing the test case into a traceable execution record; The report generation module 505 is used to output a consistency evaluation report of the simulation terminal and the actual terminal in the target scenario based on the traceable execution record.

[0122] Optionally, the test case generation module includes: The first data generation submodule is used to read the scene description file and related dependency files corresponding to the scene description information according to the scene description information, and to perform normalization processing on the scene description file and the related dependency files to obtain a normalized scene description file. The second data generation submodule is used to generate a summary and / or hash value corresponding to the normalized scene description file to obtain the scene identity identifier; The third data generation submodule is used to determine the feasible representation corresponding to the target scene based on the normalized scene description file, and generate the corresponding feasible hash value. The fourth data generation submodule is used to solidify the coordinate system of the target scene, unify the units in the target scene, record the coordinate transformation relationship of the target scene, and generate the header of the test case based on the normalized scene description file. The fifth data generation submodule is used to determine the test case parameters of the test case according to the test case specification template. The test case parameters include at least the starting pose and tolerance of the target to be tested in the target scenario, the target definition and success criteria, the instruction text, the passage constraints and termination conditions, the indicator statistical standards, the alignment anchor point trigger criteria and the recording specifications. The test case generation submodule is used to generate the test cases based on the scenario identity identifier, the passable hash value, the test case header, and the test case parameters.

[0123] Optionally, the instruction generation module includes: The observation data reading submodule is used to read the observation data contained in the decision request; The instruction generation submodule is used to generate the structured intermediate instructions based on the test cases and the observation data.

[0124] Optionally, the instruction generation submodule includes: The target determination submodule is used to determine the specified start and end positions in the target scenario based on the test cases. The location determination submodule is used to determine the location and attitude of events and the target to be measured in the target scene based on the observation data. The instruction determination submodule is used to generate corresponding window parameters and action parameters based on the starting position, the ending position, the events in the target scene, the position and attitude of the target to be tested, and a preset navigation strategy, so as to obtain the structured intermediate instruction.

[0125] Optionally, the data recording module includes: The first recording submodule is used to record the instruction identifier, instruction sequence number, execution window number, timestamp and / or step sequence, and observation summary of any instruction generated into the traceable execution record; The second recording submodule is used to record the window start boundary, window end boundary, and execution summary of the simulation terminal and the actual terminal when executing the test case to the traceable execution record; The third recording submodule is used to record the corresponding event that triggers the decision request to the traceable execution record; The associated record submodule is used to associate and record related data corresponding to the same round of decisions.

[0126] Optionally, the report generation module includes: The data reading submodule is used to read the traceable execution record and to read the first related data and the second related data. The alignment processing submodule is used to align the first related data and the second related data, and determine the difference location conclusion between the first related data and the second related data; The report generation submodule is used to obtain the consistency assessment report based on the difference location conclusion.

[0127] The step of aligning the first relevant data and the second relevant data to determine the difference location conclusions between the first relevant data and the second relevant data includes: The difference point determination submodule is used to determine the difference points between the simulation end and the actual machine end in the corresponding window based on the alignment anchor points, window information and step sequence recorded in the first relevant data and the second relevant data; The difference localization conclusion determination submodule is used to obtain the difference localization conclusion based on the execution summary of the window where the difference point is located.

[0128] The consistency traceability device between simulation testing and actual testing in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in a terminal. This device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be decision-making terminals, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0129] The consistency traceability device between simulation testing and actual testing in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0130] The consistency traceability device between simulation testing and actual testing provided in this application embodiment can achieve... Figures 2 to 4 The various processes implemented by the consistency traceability device between simulation testing and actual testing in the method embodiment will not be described again here to avoid repetition.

[0131] Optionally, this application embodiment also provides an electronic device, including a processor 110, a memory 109, and a program or instructions stored in the memory 109 and executable on the processor 110. When the program or instructions are executed by the processor 110, they implement the various processes of the above-described simulation test and real machine test consistency traceability method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0132] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0133] Figure 6 This is a schematic diagram of the hardware structure of an electronic device proposed in an embodiment of this application. The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0134] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here. This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described simulation test and real machine test consistency traceability method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0135] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0136] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described simulation test and real machine test consistency traceability method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0137] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0138] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, decision-making terminal, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0140] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method of tracing the consistency of simulation tests and real machine tests, characterized by, The method includes: Generate test cases corresponding to the target scenario based on the scenario description information and the test case specification template; The test cases are simultaneously distributed to both the simulation and the actual machine. When a decision request is issued by either the simulation terminal or the physical terminal, a structured intermediate instruction is generated based on the test case and the decision request to drive the simulation terminal or the physical terminal to execute the test case in the corresponding execution window. The first relevant data generated by the simulation terminal when executing the test case and the second relevant data generated by the real machine terminal when executing the test case are associated and recorded in the traceable execution record; Based on the traceable execution records, a consistency evaluation report of the simulation terminal and the actual terminal in the target scenario is output.

2. The method of claim 1, wherein, The process of generating test cases based on scenario description information and test case specification templates includes: Based on the scene description information, read the scene description file and related dependency files corresponding to the scene description information, and perform normalization processing on the scene description file and related dependency files to obtain the normalized scene description file; Generate a summary and / or hash value corresponding to the normalized scene description file to obtain the scene identity identifier; Based on the normalized scene description file, determine the traversable representation corresponding to the target scene and generate the corresponding traversable hash value; Based on the standardized scene description file, the coordinate system of the target scene is fixed, the units in the target scene are unified, the coordinate transformation relationship of the target scene is recorded, and the header of the test case is generated. The test case parameters are determined according to the test case specification template. The test case parameters include at least the starting pose and tolerance of the target under test in the target scenario, the target definition and success criteria, the instruction text, the passage constraints and termination conditions, the indicator statistical standards, the alignment anchor point trigger criteria and the recording specifications. The test cases are generated based on the scenario identity identifier, the passable hash value, the test case header, and the test case parameters.

3. The consistency traceability method between simulation testing and actual testing according to claim 1, characterized in that, Based on the test cases and the decision request, generate structured intermediate instructions, including: Read the observation data contained in the decision request; The structured intermediate instructions are generated based on the test cases and the observation data.

4. The consistency traceability method between simulation testing and actual testing according to claim 3, characterized in that, Based on the test cases and the observation data, the structured intermediate instructions are generated, including: Based on the test cases, determine the specified start and end positions in the target scenario; Based on the observation data, determine the events in the target scene, the position and attitude of the target to be measured; Based on the starting point position, the ending point position, the events in the target scene, the position and attitude of the target to be tested, and a preset navigation strategy, corresponding window parameters and action parameters are generated to obtain the structured intermediate instructions.

5. The consistency traceability method between simulation testing and actual testing according to any one of claims 1-4, characterized in that, The step of associating and recording the first relevant data generated by the simulation terminal when executing the test case with the second relevant data generated by the real machine terminal when executing the test case into the traceable execution record includes: In the event of generating any instruction, the instruction identifier, instruction sequence number, execution window number, timestamp and / or step sequence, and observation summary of the instruction are recorded in the traceable execution record; The window start boundary, window end boundary, and execution summary of the simulation terminal and the actual terminal when executing the test case are respectively recorded in the traceable execution record; The corresponding event that triggers the decision request will be recorded in the traceable execution record; Related data from the same round of decision-making will be linked and recorded.

6. The consistency traceability method between simulation testing and actual testing according to any one of claims 1-4, characterized in that, The step of outputting a consistency evaluation report of the simulation terminal and the actual terminal in the target scenario based on the traceable execution record includes: Read the traceable execution record, and read the first relevant data and the second relevant data; Align the first relevant data with the second relevant data to determine the difference location conclusion between the first relevant data and the second relevant data; Based on the findings of the difference localization, the consistency assessment report is obtained.

7. The consistency traceability method between simulation testing and actual testing according to claim 6, characterized in that, The step of aligning the first relevant data and the second relevant data to determine the difference location conclusions between the first relevant data and the second relevant data includes: Based on the alignment anchor points, window information, and step sequence recorded in the first and second related data, the differences between the simulation end and the actual machine end in the corresponding window are determined; The difference localization conclusion is obtained based on the execution summary of the window where the difference point is located.

8. A consistency traceability system between simulation testing and actual machine testing, characterized in that, The system includes: The consistency test case generation module is used to generate test cases corresponding to the target scenario based on the scenario description information and the test case specification template, and to distribute the test cases to the inference server module and the client execution module. The inference server module is used to receive the decision request sent by the client execution module, generate the corresponding structured intermediate instructions based on the test cases and the observation data in the decision request, and combine them with the preset navigation strategy, and send the structured intermediate instructions to the client execution module. The client execution module is used to control the simulation terminal and the actual terminal to execute the test cases according to the structured intermediate instructions, and send the first relevant data generated by the actual terminal and the second relevant data generated by the simulation terminal to the traceable record and regression comparison module. The data transmission and recording association module is used to transmit observation data and structured intermediate instructions between the inference server module and the client execution module, and write association fields such as instruction identifier and output sequence number into the traceable execution record; The traceable record and regression comparison module is used to save the first related data and the second related data, align the first related data and the second related data according to the alignment anchor point, window information and step sequence, and output a consistency assessment report.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of any of the methods described in claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of any of the methods described in claims 1-7.

Images