Ai system joint debugging verification method and system

Abstract

The application discloses an AI system joint debugging verification method and system, relates to the technical field of software testing, and comprises the following steps: a test platform acquires execution state identifiers generated when an AI subsystem executes test cases in different rounds and result files derived in two rounds; when the number of entries with a successful execution result state is less than the total number of test cases, a missing test case list is generated according to the execution state identifiers with a failed execution result state; content attribution verification is performed on each entry in the second result file, suspected misposition entries are marked, boundary consistency judgment is performed in combination with the missing test case list, and misposition boundary identifiers containing starting entry position data are obtained; a report suspension instruction is generated in response to the misposition boundary identifiers, the entries in the second result file are re-paired with the entries with the same test case number in the first result file according to a test case number mapping relationship table, and a difference report is generated, so that false differences caused by entry misposition are eliminated, and the work of manually positioning misposition boundaries is saved.

Description

Technical Field

[0001] This application relates to the field of software testing technology, and in particular to AI system integration and verification methods and systems. Background Technology

[0002] During the integration and testing of AI systems, the testing platform needs to assign a list of test cases to the AI ​​subsystems. The AI ​​subsystems then execute the test cases in rounds and export result files after each round. The comparison engineer compares the first result file exported after executing all test cases in the first round with the second result file exported after executing the second round, line by line. Based on the line number-aligned entries, a difference report is generated, and the impact of version changes on the output of the AI ​​subsystems is determined based on the difference report.

[0003] In the aforementioned joint debugging and comparison scenario, the execution of test cases by the AI ​​subsystem is susceptible to network interruptions. If no result is received from the AI ​​subsystem within a preset waiting time after an execution request is sent, a network interruption event occurs, and the test case is deemed to have failed. Failed test cases are not written to the result file, resulting in a smaller number of entries in the second result file compared to the total number of test cases in the test case list. With missing entries, all entries following the missing test case are shifted forward, causing a misalignment between line numbers and test case numbers in the second result file. When the comparison platform compares line by line according to line number, it misjudges many entries that are essentially identical as differing entries, resulting in a discrepancy report filled with false differences caused by misalignment, rendering the discrepancy conclusions meaningless.

[0004] To pinpoint the misalignment, the comparison engineer needs to manually check each entry in the second result file, confirming the relationship between the entry's content and the test case number, and then calculating the starting line and number of misaligned lines. When the test case list contains hundreds of test cases, manual verification is time-consuming, and the starting line of the misalignment is easily missed. Furthermore, the generation and archiving of difference reports are usually automated; error reports are archived and displayed before engineers discover the problem, requiring subsequent corrections to re-execute the comparison process, further extending the integration testing cycle. Summary of the Invention

[0005] In view of the aforementioned problems, this application is hereby filed.

[0006] Therefore, this application provides an AI system integration and verification method and system, which can solve the problem that test case execution failures lead to missing entries and misaligned line numbers in the result file, resulting in a large number of false differences in the difference report generated by line-by-line comparison, and the misalignment boundary needs to be manually located.

[0007] To solve the above-mentioned technical problems, this application provides the following technical solution: Firstly, this application provides a method for verifying and debugging an AI system, including: obtaining the execution status identifier generated when the AI ​​subsystem executes test cases in rounds, and the first result file exported after executing all test cases in the test case list in the first round and the second result file exported after executing test cases in the second round; the execution status identifier includes test case number data, execution result status data, and round identifier data; In response to the comparison command, when the number of entries with a successful execution result in the execution status identifier corresponding to the current round identifier data is less than the total number of use cases in the use case list, use case number data is extracted based on the execution status identifier with a failed execution result, and a list of missing use cases is generated. For each entry in the second result file, a content attribution check is performed. The content features of the entry are matched with the input features corresponding to the test case number mapped to the current line number of the entry. Entries with inconsistent content features are marked as suspected misaligned entries. Boundary consistency judgment is performed based on the missing test case list and the suspected misaligned entries to obtain the misaligned boundary identifier containing the starting entry position data. In response to the misalignment boundary marker, a report pause command is generated. The second result file entries are re-paired with the entries with the same use case number in the first result file according to the use case number mapping table, and a difference report is generated. The use case number mapping table records the correspondence between each use case number in the use case list and the positions of the first and second result file entries.

[0008] Preferably, the execution status identifier generated when the AI ​​subsystem executes test cases in rounds includes: At the end of each test case execution in the AI ​​subsystem, an execution status identifier is generated and written to the record table. The execution status identifier also includes execution time data, interruption association identifier data, and confirmation status data. The interruption association identifier data indicates whether a network interruption occurred during the execution of the test case. A network interruption is the state in which no result is received from the AI ​​subsystem within a preset waiting time after the execution request is sent. For test cases whose execution result status is "execution failed", an execution status identifier is also generated and written to the record table when the execution ends; The execution status identifiers generated in each round are collected into a record table according to the round identifier data to obtain the execution status identifiers generated when the AI ​​subsystem executes test cases in rounds.

[0009] Preferably, when the number of entries with a successful execution result in the execution status identifier corresponding to the current round identifier data is less than the total number of use cases in the use case list, use case number data is extracted based on the execution status identifier with a failed execution result to generate a missing use case list, including: Determine whether the number of entries with a successful execution result among all execution status identifiers corresponding to the current round identifier data is equal to the total number of use cases in the use case list; If the number is less than the total number of test cases, extract the test case number data based on the execution status identifier of the execution result status being failed, and generate a list of missing test cases by arranging them in ascending order of test case number. Send an integrity alert signal containing the test case numbers from the missing test case list to the comparison engineer's terminal; If the comparison engineer selects to continue the comparison via the confirmation button on the terminal, the list of missing test cases will be bound to the current comparison task; if the engineer selects to pause the comparison, the comparison task will be suspended and the list of missing test cases will be retained; if no button response is received within the preset waiting time, the comparison will continue and an interaction record indicating that no integrity confirmation has been received will be generated, thus completing the generation of the list of missing test cases.

[0010] Preferably, the step of performing content attribution verification on each entry in the second result file, matching the entry content features with the input features corresponding to the test case number mapped to the current line number of the entry, and marking entries with mismatched content features as suspected misalignment entries includes: For each entry in the second result file, extract the content features that reflect the input category and data fields in the entry content; Match the entry content features with the input features registered in the use case list for the use case number mapped to the current line number of the entry; If the content features of an entry do not match the registered input features, the entry will be marked as a suspected misalignment entry. If the content characteristics of an entry match the registered input characteristics, the entry is marked as a normal entry, and the content ownership verification of each entry in the second result file is completed.

[0011] Preferably, marking entries with inconsistent content features as suspected misalignment entries includes: For entries marked as suspected misalignments, the entry content features are compared with the entry content features of the use case number plus one in the first result file; If they match, determine that a single line is misaligned and record the number of misaligned lines; If they do not match, continue to compare the content features with the entries in the first result file whose test case number is two or three, until an entry with matching content features is found or the comparison reaches the end of the preset adjacent range. When a matching entry is found, record the number of misaligned lines. If the content characteristics of an entry do not match the content characteristics of any adjacent entry in the first result file, the entry is determined to be an output content anomaly, a content anomaly identifier is generated, a content anomaly review request is sent to the execution engineer terminal, and the mark of the suspected misaligned entry is completed.

[0012] Preferably, the step of performing boundary consistency judgment based on the missing use case list and the suspected misaligned items to obtain the misalignment boundary identifier containing the starting item position data includes: The number of the first failed use case is determined based on the missing use case list, and the position of the first failed use case number in the first result file is determined based on the use case number mapping table, which serves as the candidate misalignment starting boundary. If the position of the first suspected misaligned entry in the first result file is consistent with the candidate misalignment start boundary, and the number of misaligned rows of the suspected misaligned entry is consistent with the number of previously accumulated missing test cases, a misalignment boundary identifier containing the start entry position data and the number of misaligned rows is generated to complete the boundary consistency judgment. If all entries in the second result file are marked as belonging to normal entries, a no-misalignment flag is generated, and the boundary consistency judgment is completed. If a suspected misaligned entry exists but its position is inconsistent with the candidate misalignment starting boundary, or the number of misaligned rows does not match the number of missing test cases, a boundary uncertainty flag is generated, and a boundary verification request is sent to the comparison engineer's terminal to complete the boundary consistency judgment.

[0013] Preferably, the step of responding to the misalignment boundary identifier, generating a report pause instruction, and re-pairing the entries in the second result file with the entries of the same use case number in the first result file according to the use case number mapping table to generate a difference report includes: In response to the misalignment boundary marker, a report pause command is generated to pause the archiving and display of the discrepancy report; Generate a realignment instruction, match the entries in the second result file with the entries in the first result file with the same use case number according to the use case number mapping table, compare the contents, leave the rows of missing use cases blank and mark them as entries to be filled; Wait for the alignment completion feedback signal after pairing is completed. If no alignment completion feedback signal is received within the preset waiting time, resend the realignment command. If no realignment is received again, send a manual alignment request to the comparison engineer terminal. Send a paused notification signal containing the pause reason and a summary of the missing use case list to the comparison engineer terminal and the execution engineer terminal respectively, and generate a difference report based on the re-paired items.

[0014] Preferably, the method further includes: In response to the alignment completion feedback signal, a missing test case rerun query is generated and sent to the execution engineer terminal. The content includes a list of missing test cases and two options: rerun the missing test cases or generate a difference conclusion based on the existing items. If the execution engineer chooses to run missing test cases, the missing test case list will be converted into a rerun task. After the rerun test cases are completed, an execution status identifier will be generated and written to the record table. The rerun results will be appended to the corresponding line in the second result file. If the engineer chooses to generate a difference conclusion based on the existing items, the missing test cases will be marked as uncompared in the difference report. After the rerun is completed, a second confirmation judgment is performed: if the items are complete after the rerun and all previously suspected misaligned items are made consistent after being paired by use case number, a false difference confirmation mark is generated and the report is released from suspension; if the items are complete after the rerun but there are still items whose content does not match the items with the same use case number in the first result file, a real difference mark is generated; if the rerun fails again, or the rerun result does not match the content characteristics of the adjacent items in either the first or second result file, a question mark is generated, a manual review request is sent to the team leader's terminal, the report is kept in suspension, and a difference report is generated according to the re-paired items.

[0015] Preferably, the method further includes: Obtain network interruption event records, which are generated when a request wait timeout state with the AI ​​subsystem is detected, and include interruption start time data and interruption end time data; The execution status identifier is segmented and verified based on the interruption period defined by the interruption start time data and the interruption end time data. The execution status identifier generated within the interruption period is not used as the basis for judging the final execution result of the test case. Instead, the test case completion receipt returned by the AI ​​subsystem is used to determine the execution success. The test case completion receipt carries the test case number and the position of the result item. Test cases for which no completion receipt has been received and whose execution request time falls within the interruption period are registered as execution failures and added to the missing test case list. If two execution status indicators overlap for the same test case, the one with the earlier completion receipt arrival time shall be used, and the other shall be marked as a duplicate record and removed. When network interruption event logs are unavailable, a query prompt is sent to the execution engineer terminal to ask whether a network interruption has occurred and the duration of the interruption. Based on the key press response, alternative interruption duration data is generated to complete the segmented verification of the execution status identifier.

[0016] Secondly, this application also provides an AI system integration and verification system, including: an execution status recording module, an integrity judgment module, a content ownership verification module, a boundary judgment module, and a report control module; The execution status recording module is used to generate an execution status identifier, which includes test case number data, execution result status data, and round identifier data, and write it into the record table when each test case of the AI ​​subsystem is executed. The integrity judgment module is used to respond to the comparison command and generate a missing use case list based on the execution status identifiers with execution result status of failure when the number of entries with successful execution result status in the current execution status identifier is less than the total number of use cases in the use case list. The content attribution verification module is used to perform content attribution verification on each entry in the second result file, matching the content features of the entry with the input features corresponding to the test case number mapped to the current line number of the entry, and marking entries with inconsistent content features as suspected misaligned entries. The boundary judgment module is used to perform boundary consistency judgment based on the missing use case list and the suspected misalignment item to obtain the misalignment boundary identifier containing the starting item position data. The report control module is used to generate a report pause instruction in response to the misalignment boundary identifier, and then re-pair the entries in the second result file with the entries with the same use case number in the first result file according to the use case number mapping table to generate a difference report.

[0017] Implementing this application will have the following beneficial effects: This application provides a method and system for AI system integration and verification. The invention automatically counts the number of successful entries and generates a list of missing test cases by using test case number data, execution result status data, and round identifier data in the execution status identifier. This eliminates the need for engineers to manually check missing entries in the second result file, shortening the confirmation time for missing test cases. Through content attribution verification, the invention matches the content features of the entries with the input features corresponding to the test case numbers mapped to the line numbers. Combined with boundary consistency judgment, the testing platform can automatically determine the starting position of the misaligned entry and the number of misaligned lines, replacing the manual calculation of misalignment boundaries. By using a report pause command to prevent the archiving of error difference reports and re-pairing entries according to the test case number mapping table, the entries in the difference report correspond one-to-one with the test case numbers, eliminating false differences caused by entry misalignment and improving the credibility of the difference conclusions. Combined with segmented verification during network interruption periods and the missing test case rerun process, the invention also reduces the interference of network interruption on execution status determination and rerun result appending, shortening the overall cycle of integration and comparison. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the AI ​​system integration and verification method involved in this application; Figure 2 This is a schematic diagram of a process segment for data acquisition and integrity judgment in the AI ​​system integration and verification method involved in this application; Figure 3 This is a schematic diagram of a process segment for content attribution verification and proximity comparison of the AI ​​system integration and verification method involved in this application; Figure 4 This is a schematic diagram of a process segment for determining the boundary consistency of the AI ​​system integration and verification method involved in this application; Figure 5 This is a schematic diagram of a process segment involving the report suspension and realignment of the AI ​​system integration and verification method involved in this application; Figure 6 This is a schematic diagram of a process segment involving supplementary inquiry and secondary confirmation for the AI ​​system integration and verification method involved in this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] Example 1, referring to Figures 1-6 This is the first embodiment of the present application, which provides an AI system integration and verification method, including: This application provides a method that can effectively solve the problems mentioned above. The following will describe in detail how to implement the AI ​​system joint debugging and verification method with multiple embodiments. Figure 1 The flowchart of the AI ​​system integration and verification method is shown, including: Step S1: Obtain the execution status identifier generated when the AI ​​subsystem executes test cases in rounds, as well as the first result file exported after executing all test cases in the test case list in the first round and the second result file exported after executing test cases in the second round; the execution status identifier includes test case number data, execution result status data, and round identifier data; It should be noted that the action of obtaining the execution status identifier is passively received by the test platform configured with this method at the end of each test case execution. At the end of each test case, the AI ​​subsystem sends back a message to the test platform carrying test case number data, execution result status data and round identifier data. The test platform writes the received messages into the record table in the order of arrival, thereby forming the execution status identifier generated when the test cases are executed in rounds. The round identifier data in the record table is used to distinguish the execution affiliation of the same test case in different rounds during subsequent statistics.

[0022] Specifically, the acquisition of the first result file takes place at the end of the first round of execution. After confirming that all test cases in the test case list have been dispatched and completed, the test platform sends a result export request to the AI ​​subsystem, receives the set of result items returned by the AI ​​subsystem in the order of the test case list, and writes the set of result items to disk as the first result file. The acquisition of the second result file is achieved after the completion of the second round of execution by triggering an export request in the same way and writing it to disk. Each item in the two result files corresponds to the output content of a test case.

[0023] Furthermore, after obtaining the two result files, the testing platform will also establish a test case number mapping table. The mapping table records the correspondence between each test case number in the test case list and the position of the first result file entry and the position of the second result file entry. The construction process of the mapping table will be explained separately in the extended content of step S3. Here, we will first provide the position index basis for content ownership verification and re-pairing.

[0024] In some embodiments, step S1, which involves obtaining the execution status identifier generated when the AI ​​subsystem executes test cases in rounds, may further include the following steps: Step S1.1: When each test case of the AI ​​subsystem is completed, an execution status identifier is generated and written to the record table. The execution status identifier also includes execution time data, interruption association identifier data, and confirmation status data. The interruption association identifier data indicates whether a network interruption occurred during the execution of the test case. A network interruption is the state that no result is received from the AI ​​subsystem within a preset waiting time after the execution request is sent. Step S1.2: For test cases whose execution result status is "execution failed", an execution status identifier is also generated and written to the record table at the end of execution; Step S1.3: Collect the execution status identifiers generated in each round into a record table according to the round identifier data to obtain the execution status identifiers generated when the AI ​​subsystem executes test cases in rounds.

[0025] Understandably, the execution status identifier is the data source for subsequent integrity judgment and difference comparison. If it is only recorded when the test case is executed successfully, the failed test cases and failure reasons will be outside the coverage of the record table, making it impossible to directly extract the list of missing test cases from the record. Therefore, it is necessary to generate a uniformly structured identifier entry at the end of the execution of each test case, regardless of whether it is successful or not.

[0026] Specifically, the actions for generating execution status identifiers include extracting five fields from the received end message, registering the use case number data as is in the use case list, registering the execution result status data as either success or failure, registering the execution time data as the timestamp of the end message returned by the AI ​​subsystem, registering the confirmation status data as a status bit indicating whether the engineer has confirmed the integrity, and generating interruption association identifier data. The interruption association identifier data is generated by starting a timer simultaneously with the execution request, incrementing the timer by seconds. When the timer reaches the interruption determination duration and no result is received from the AI ​​subsystem, the interruption is disabled. The associated identifier data is registered as an interrupted value, and the timer is reset when a return result is received. When a return result is received within the interruption determination time, the interrupted associated identifier data is registered as an uninterrupted value. The interruption determination time is used to define the upper limit of the time after the request is sent without receiving a return result. It is obtained by statistically analyzing the maximum time taken for the AI ​​subsystem to return the result of a single use case in historical operation. In this embodiment, the interruption determination time is preferably 30 seconds. In practical applications, it can be adjusted within the range of 10 seconds to 60 seconds. For generation-type use cases with longer response times, a value closer to the upper limit is used, and for query-type use cases with shorter response times, a value closer to the lower limit is used.

[0027] It should be noted that for test cases whose execution result status is "execution failed", the action of generating execution status identifiers is exactly the same as that for successful test cases. That is, the test case number data, execution time data, interruption association identifier data, confirmation status data and round identifier data are extracted and written into the record table. The only difference is that the execution result status data is registered as a failure value. This ensures that the number of entries in the record table corresponds one-to-one with the total number of test cases in the test case list in the round dimension. Any test case can be retrieved with a unique identifier entry in the record table in any round.

[0028] Furthermore, the action of aggregating the execution status identifiers generated in each round into a record table is accomplished by setting a round field in the record table. When each identifier entry is written, the round number issued when the test case was dispatched in this round is filled into the round field. After aggregation, the record table supports filtering all identifier entries in this round by the value of the round field, thereby obtaining the execution status identifiers generated by the AI ​​subsystem when executing test cases in rounds. The filtering method is to scan the record table one by one, and copy the entries whose round field value is equal to the target round number to the statistical set of this round. The statistical set retains the original writing order of the entries in the record table.

[0029] In practical applications, in addition to the fields mentioned above, the record table also retains a write sequence number field. The write sequence number is auto-incremented according to the arrival order, which is used to distinguish the order when multiple identifiers with the same use case number overlap. The handling method for overlapping situations is explained in the extended content of steps S1.4 to S1.8.

[0030] In this embodiment, the initial value of the confirmed status data is registered as unconfirmed. The integrity judgment module updates the status bit after the comparison engineer completes the key response. The update of the status bit does not change the content of other fields, so as to maintain the stable searchability of the identification item in the round dimension.

[0031] Understandably, through the above field composition and aggregation method, each execution status identifier in the record table carries not only the three essential comparison data (use case number, execution result, and round), but also the three auxiliary data (execution time, interruption association, and confirmation status), providing a complete source of original data for the automatic generation of the missing use case list and the segmented verification of network interruption periods.

[0032] For example, the test case list contains 10 test cases, numbered 1 to 10. After the second round of dispatch, test cases numbered 1 to 6 generate six identifier entries upon completion of their respective executions. The execution result status data is all recorded as successful, and the interruption-related identifier data is all recorded as uninterrupted. The round number field is filled with 2. Test case number 7 does not receive a return result within 30 seconds after issuing the execution request. The interruption-related identifier data is recorded as interrupted, and the execution result status data is recorded as failed. Identifier entries are still generated and written to the record table. Test cases numbered 8 to 10 end normally and generate their own identifier entries. After writing, the record table obtains ten identifier entries under the filtering condition that the round number field is 2, which constitute the execution status identifier set of the second round.

[0033] In some preferred embodiments, step S1, which involves obtaining the execution status identifier generated when the AI ​​subsystem executes test cases in rounds, may further include the following steps: Step S1.4: Obtain network interruption event records. The network interruption event records are generated when a request wait timeout state is detected with the AI ​​subsystem, and include interruption start time data and interruption end time data. Step S1.5: Using the interruption period defined by the interruption start time data and the interruption end time data as the boundary, perform segmented verification on the execution status identifier. The execution status identifier generated within the interruption period is not used as the basis for judging the final execution result of the test case. Instead, the test case completion receipt returned by the AI ​​subsystem is used to determine the successful execution. The test case completion receipt carries the test case number and the position of the result item. Step S1.6: Test cases that have not received a test case completion receipt and whose execution request time falls within the interruption period are registered as execution failures and added to the missing test case list; Step S1.7: If two execution status identifiers overlap for the same test case, the one with the earlier test case completion receipt arrival time shall be used, and the other one shall be marked as a duplicate record and removed. Step S1.8: When the network interruption event record is unavailable, send an inquiry prompt to the execution engineer terminal asking whether a network interruption has occurred and the interruption period. Generate alternative interruption period data based on the key response and complete the segmented verification of the execution status identifier.

[0034] It should be noted that the acquisition of network outage event records depends on the monitoring of request waiting on the link between the test platform and the AI ​​subsystem. Specifically, when a request is sent, the time of the request is recorded and a waiting timer is started. When the waiting timer reaches the timeout period and no return data packet is received, a request waiting timeout state is determined, and the time of the request is recorded as the outage start time data. Subsequently, lightweight probe packets are continuously sent to the AI ​​subsystem. When a probe packet receives a response within the timeout period, the time of receiving the response is recorded as the outage end time data. The outage start time data and the outage end time data together constitute a network outage event record.

[0035] Specifically, the execution status identifier is segmented and verified based on the interruption period. This involves reading the execution time data for each identifier entry in the record table and comparing it with the interruption start time and end time data in each network interruption event record. If the execution time data falls within the closed interval defined by the start and end times, the identifier entry is registered as an entry within the interruption period. If the execution time data falls outside all interruption periods, the identifier entry is retained as a normal entry. For identifiers registered as entries within the interruption period, their execution result status data is no longer used as the basis for judging the final execution result of the test case. Instead, the judgment basis is to check whether a test case completion receipt has been received from the AI ​​subsystem. The test case completion receipt is reissued by the AI ​​subsystem after the test case is actually executed. The receipt contains the test case number and the position of the output entry of the test case in the result file. When the receipt is received, the corresponding test case is judged as successfully executed, and the attribution information in the record table is updated according to the position of the result entry carried by the receipt.

[0036] Furthermore, for test cases that have not received a test case completion receipt and whose execution request time falls within the interruption period, the actions registered as execution failures include locating the identifier entry corresponding to the test case number in the record table, rewriting the execution result status data to a failure value, and appending the test case number to the temporary list of the missing test case list. The appending order is arranged according to the execution request time, thereby ensuring the order of the missing test case list in the number dimension, which facilitates the subsequent extraction of the first failed test case number.

[0037] It should be noted that if two execution status identifiers overlap for the same test case, it is usually triggered concurrently by the AI ​​subsystem resending results and timeout judgment before and after the interruption ends. The handling method is to extract the receipt arrival time of the two identifier entries under the same test case number, keep the one with the earlier arrival time as a valid entry, mark the one with the later arrival time as a duplicate record and remove it from the statistical set of this round. The removal action only applies to the statistical set, and the original entry in the record table retains the mark for traceability.

[0038] Understandably, the availability of network interruption event records is confirmed through integrity checks. The check involves reading the start and end times from the records. If both exist and the end time is greater than the start time, the record is considered available. If either time data is missing or there is a contradictory value where the end time is less than or equal to the start time, the record is considered unavailable. When an unavailable record is determined, a prompt is sent to the execution engineer's terminal. The prompt includes an option button for whether an interruption occurred and input fields for the interruption start and end times. Based on the button response and the values ​​received from the input fields, alternative interruption period data is generated. This alternative interruption period data has the same data structure as the normally acquired network interruption event records. After generation, it replaces the original record and participates in the segmentation verification, thereby completing the segmentation verification of the execution status identifier.

[0039] In practical applications, segmented verification is performed once before the response to each round of comparison instructions. The verification scope is limited to the identifier entries whose round field value is equal to that of the current round. Identifier entries of other rounds are not modified to avoid data interference across rounds.

[0040] The timeout determination duration and the interruption determination duration in step S1.1 use the same value source, that is, it is derived from the statistics of the maximum response time of the AI ​​subsystem returning results under historical uninterrupted operation. In this embodiment, the preferred timeout determination duration is 20 seconds. In actual applications, it can be adjusted within the range of 10 seconds to 45 seconds. A larger value is used in deployment environments with large fluctuations in link quality, and a smaller value is used in deployment environments with stable links.

[0041] For example, in the second round, test case number 7 issued an execution request at 10:05:00. No response data packet was received within 20 seconds, indicating a request timeout. The interruption start time was recorded as 10:05:00. At 10:06:10, a response was received, and the interruption end time was recorded as 10:06:10, resulting in an interruption period from 10:05:00 to 10:06:10. Test cases numbered 7 and 8 both fell within this interruption period. Segmented verification registered both identifiers as entries within the interruption period. Test case number 7 did not receive a completion receipt, so it was registered as an execution failure and added to the temporary list of missing test cases. Test case number 8 received a completion receipt with the result entry location at 10:05:50, and its execution was changed to successful. At the same time, the AI ​​subsystem reissued a result, causing test case number 8 to have two identifiers. After comparing the receipt arrival times, the earlier one was retained, and the other was marked as a duplicate and removed.

[0042] Step S2: In response to the comparison command, when the number of entries with a successful execution result in the execution status identifier corresponding to the current round identifier data is less than the total number of use cases in the use case list, extract the use case number data based on the execution status identifier with a failed execution result and generate a missing use case list.

[0043] Understandably, the comparison command is initiated by the comparison engineer on the terminal. The command content carries the target round number. After receiving the command, the test platform first uses the target round number as the filtering condition to locate all the identification entries of this round in the record table, and then performs a count of the number of successful entries on the location results to avoid including identification entries from other rounds in the integrity judgment of this round.

[0044] Specifically, the statistical action for the number of successful entries includes scanning the statistical set for this round one by one, taking the execution result status data as the count of successful entries and accumulating it to obtain the number of successful entries. The total number of test cases is directly read from the field of the number of test case entries registered in the test case list. The comparison result of the two values ​​determines the direction of the subsequent branch. When the number of successful entries is equal to the total number of test cases, the generation of the missing test case list is skipped and the comparison is directly entered. When it is less than the total number of test cases, the extraction action is triggered.

[0045] Furthermore, the action is extracted from the statistical set of this round and the execution result status data is filtered to identify entries with the value of failure. The use case number data is read one by one to form a number set. After sorting, the number set is output as a list of missing use cases. The specific steps of sorting and list generation are explained in the extended content of steps S2.1 to S2.4.

[0046] In this application, step S2, when the number of entries with a successful execution result in the execution status identifier corresponding to the current round identifier data is less than the total number of use cases in the use case list, extracts use case number data based on the execution status identifiers with a failed execution result to generate a missing use case list, may further include the following steps: Step S2.1: Determine whether the number of entries with a successful execution result among all execution status identifiers corresponding to the current round identifier data is equal to the total number of use cases in the use case list; Step S2.2: If the number of test cases is less than the total number of test cases, extract the test case number data based on the execution status identifier of the execution result being failed, and generate a list of missing test cases by arranging them in ascending order of test case number; Step S2.3: Send an integrity prompt signal containing the test case numbers from the missing test case list to the comparison engineer's terminal; Step S2.4: If the comparison engineer selects to continue the comparison via the confirmation button on the terminal, the missing test case list will be bound to the current comparison task; if the engineer selects to pause the comparison, the comparison task will be suspended and the missing test case list will be retained; if no button response is received within the preset waiting time, the comparison will continue and an interaction record indicating that no integrity confirmation has been received will be generated, thus completing the generation of the missing test case list.

[0047] It should be noted that the determination of the number of successful entries is performed internally by a counter within the integrity determination module. Specifically, the success counter is initialized to 0. The statistical set for this round is read one by one in the order of writing. Whenever an entry with a successful status data value is read, the success counter is incremented by one. After the scan is completed, the current value of the success counter is read and compared with the total number of test cases registered in the test case list. When the two are equal, the entries for this round are considered complete. When the success counter value is less than the total number of test cases, it is determined that there are missing entries and the process is transferred to the extraction branch.

[0048] Specifically, the execution method for extracting branches is to traverse the statistical set of the current round again, filter out the entries whose execution result status data value is failure, read the test case number data from each selected entry, and write the read test case number into the temporary storage sequence; then sort the temporary storage sequence by scanning in one pass. In each pass, the numerical values ​​of two adjacent test case numbers are compared. If the previous number is greater than the next number, their storage positions in the temporary storage sequence are swapped. The scanning is repeated until no swaps occur in a certain pass, resulting in an ordered sequence arranged in ascending order of test case numbers. A missing test case list is generated based on this ordered sequence. In addition to the list of numbers, the list also records the time of list generation and the round number to which it belongs.

[0049] Furthermore, the integrity prompt signal is generated and sent by assembling a message containing all test case numbers in the missing test case list, the round number to which the list belongs, and two option button identifiers. This message is then pushed to the terminal interface via the message channel between the test platform and the comparison engineer's terminal. After the message is pushed, a response timer is started on the test platform side.

[0050] Understandably, the three possible responses to a button are determined by a response timer and the button message. When the comparison engineer presses the "Continue Comparison" button, the testing platform receives a button message carrying the "Continue" value, writes the list number of the missing test cases into the binding field of this comparison task, and associates the missing test case list with this comparison task. When the "Pause Comparison" button is pressed, a button message carrying the "Pause" value is received, the status field of the comparison task is rewritten to the "Suspended" value, the comparison task is removed from the execution queue, and the missing test case list is retained for use when the task resumes. When the response timer reaches the response waiting time and no button message is received, the "Continue" button is pressed. The comparison process continues, and an interaction record indicating that no integrity confirmation has been received is generated. The interaction record registers the task number, the start and end times of the wait, and the conclusion of continuing the process, and is written to the interaction log table for storage. The response waiting time is used to limit the upper limit of the waiting time for the comparison engineer to respond to the integrity prompt signal. It is obtained by statistically analyzing the historical response time distribution of the comparison engineer's response integrity prompts during the joint debugging test. In this embodiment, the preferred response waiting time is 5 minutes. In practical applications, it can be adjusted within the range of 2 minutes to 15 minutes. A smaller value is used in the deployment environment with a tight joint debugging window, and a larger value is used for comparison tasks that require sufficient manual confirmation.

[0051] In practical applications, after the binding action is completed, the list number of the missing use case list is passed down level by level in the subsequent content ownership verification and boundary consistency judgment along with the comparison task, so that each link can read the use case number in the list as needed.

[0052] In this embodiment, regardless of which path is entered, the numbered list content of the missing use case list itself remains unchanged. The three paths only change the association state between the list and the comparison task, as well as the execution state of the comparison task.

[0053] For example, after the second round of statistical set scanning is completed, the success counter value is 198, the total number of test cases is 200, and it is determined that there are missing entries; the statistical set is traversed to extract two failure identifiers, number 75 and number 76, which are written into the temporary sequence and then scanned in successive passes to obtain an ordered list of 75 and 76, generating a list of missing test cases and recording the list generation time and round number 2; the integrity prompt signal carrying 75, 76 and two option buttons is pushed to the comparison engineer's terminal. After 3 minutes, the comparison engineer presses the continue comparison button, and the test platform writes the list number into the binding field of this comparison task, completing the binding of the missing test case list with this comparison task.

[0054] Step S3: Perform content attribution verification on each entry in the second result file, match the content features of the entry with the input features corresponding to the test case number mapped to the current line number of the entry, mark entries with inconsistent content features as suspected misaligned entries, and perform boundary consistency judgment based on the missing test case list and suspected misaligned entries to obtain the misaligned boundary identifier containing the starting entry position data. It should be noted that the rule of mapping the current line number of an entry to the use case number is based on the use case number mapping relationship table. The mapping relationship table is constructed after the two result files are exported. The construction process of the mapping relationship table is fully described in step S3.0 as an independent preparation step.

[0055] Specifically, the content attribution verification is performed on each entry in the second result file. During verification, the line number of the entry is used as the key to look up the use case number to which the line number is assumed to belong in the mapping table. Then, the input features registered by the use case number are read from the use case list. At the same time, the content features are extracted from the text of the entry. The specific steps for comparing the two sets of features field by field are explained in the extended content of steps S3.1 to S3.4. The comparison result determines whether the entry is marked as a suspected misplacement entry or belongs to a normal entry.

[0056] Furthermore, the boundary consistency judgment is initiated after the missing use case list and the marking results of the suspected misaligned items are complete. The judgment process jointly checks the first failed use case number determined by the missing use case list with the position and misaligned row number of the suspected item. When the check passes, a misaligned boundary identifier containing the starting item position data and the misaligned row number data is output. When the check fails, a boundary uncertainty identifier is output. The details of generating the two identifiers are explained in the extended content of steps S3.5 to S3.8.

[0057] Step S3: After obtaining the first result file and the second result file, construct a use case number mapping table. The use case number mapping table records the correspondence between each use case number in the use case list and the position of the first result file entry and the position of the second result file entry.

[0058] It should be noted that the use case number mapping table is a shared location index for the three stages of content ownership verification, boundary consistency judgment and re-pairing. It is built after both result files are written to disk and before the comparison command is responded to. The construction action is to traverse the two result files and register the entry position according to the use case number. After the construction is completed, the mapping table supports two access methods: forward retrieval by use case number as the key and reverse retrieval by line number as the key.

[0059] Specifically, the mapping table is constructed using two scans. The first scan targets the first result file, assigning an incrementing entry position number to each entry according to the order of entries in the file. Simultaneously, it reads the use case number registered in the first field of each entry and writes the use case number and the entry position in the first result file into the first set of registration columns of the mapping table. The second scan targets the second result file, assigning an entry position number to each entry in the same way and reading the use case number. The use case number and the entry position in the second result file are then written into the second set of registration columns of the mapping table. After the two scans are completed, each row of the mapping table registers three fields: a use case number, the entry position of that number in the first result file, and the entry position of that number in the second result file.

[0060] Furthermore, when a test case number is not hit in the second scan due to missing entries in the second result file, the entry position field of the second result file for that number in the mapping table is registered as null. The null field is used as the basis for determining missing test cases in the subsequent pairing process. When an entry misalignment occurs, the entry position of each number in the second result file is registered accurately according to the actual line number when exported. The position deviation caused by the misalignment is detected by the subsequent content attribution verification. The mapping table itself does not make any corrections.

[0061] In practical applications, after the mapping relationship table is constructed, a self-check is required. The self-check method is to count whether the number of non-empty entries in the first group of registration columns is equal to the total number of use cases in the use case list. If all use cases in the first round corresponding to the first result file are executed successfully, the number of non-empty entries should be equal to the total number of use cases. If they are not equal, a construction error is registered and the comparison process is stopped. The mapping relationship table is reconstructed after the first result file is re-exported.

[0062] In this embodiment of the application, step S3 performs content attribution verification on each entry in the second result file, matches the entry content features with the input features corresponding to the use case number mapped to the current line number of the entry, and marks entries with inconsistent content features as suspected misalignment entries. This step may also include the following steps: Step S3.1: For each entry in the second result file, extract the content features that reflect the input category and data fields in the entry content; Step S3.2: Match the entry content features with the input features registered in the use case list for the use case number mapped to the current line number of the entry; Step S3.3: If the content features of an entry do not match the registered input features, mark the entry as a suspected misalignment entry; Step S3.4: If the content features of an entry match the registered input features, mark the entry as a normal entry and complete the content ownership verification of each entry in the second result file.

[0063] Understandably, content features are comparable information in the text of an entry that reflects which type of input use case the entry originates from. Since the output content of an AI subsystem usually inherits the input category and data field composition of the input use case, even if the output text changes, the input category identifier and field name are still retained in the entry. Therefore, content features can be used as a basis for determining the ownership of an entry.

[0064] Specifically, both content features and input features are represented by feature vector data structures in this application. The feature vector consists of input category elements and field name elements arranged in a fixed order. The input category elements are the input type identifier of the use case, and the field name elements are arranged in the order of appearance in the entry or use case registration information. The comparability of the two feature vectors is based on the dual correspondence of the number of elements and the order of elements.

[0065] Furthermore, the content feature extraction action is performed line by line for each entry. The text of the current entry in the second result file is read, and the text is divided into several field segments according to the field segmentation rules. Two types of information are extracted from the field segments: one is input category information, namely the input type identifier or input source mark registered in the entry; the other is data field composition information, namely the sequence of field names appearing in the entry and the order in which each field name appears. The input category information is registered as the first element of the feature vector, and the sequence of field names is registered as the subsequent elements of the feature vector according to the order in which they appear, thus obtaining the content feature vector of the entry. The field segmentation rules are obtained by statistically analyzing the fixed format of the AI ​​subsystem output entries in the historical exported files. In this embodiment, it is preferred that the field segmentation rules are composed of fixed delimiters and fixed field order. In practical applications, the field segmentation rules can be updated synchronously with the template version of the output format.

[0066] It should be noted that the matching action performs an element-by-element comparison between the content feature vector and the input feature vector. The input feature vector is read from the input features registered in the use case list. During the use case list compilation phase, the input features are pre-registered as feature vectors isomorphic to the content features according to the input category and data fields of each use case. The comparison method is to compare the values ​​one by one in the order of elements. When the number of elements in the content feature vector and the input feature vector are equal and the value of each corresponding element is consistent, it is determined that the content feature matches the registered input feature. When the value of any corresponding element is inconsistent or the number of elements is unequal, it is determined that the content feature does not match the registered input feature.

[0067] Furthermore, for entries determined to be inconsistent, a suspected misalignment value is registered for the row number of the entry in the content attribution verification result table, forming a suspected misalignment entry. At the same time, the row number and content feature vector of the entry are registered in the temporary storage area for subsequent misalignment row number comparison and boundary consistency judgment. For entries determined to be consistent, a normal attribution value is registered in the result table, forming a normal attribution entry. The two types of tags are stored mutually exclusively in the result table.

[0068] In practical applications, the matching result statistics counter is accumulated synchronously with the verification process, recording the number of suspected misaligned entries and the number of normal entries respectively. When the verification traversal ends, the two count values ​​are read and used as the basis for determining the absence of misalignment markers in steps S3.5 to S3.8.

[0069] In this application, suspected misaligned entries are only marked in the result table without modifying the original content of the second result file. The entry text and line numbers of the second result file remain as they were when exported. All attribution determination information is centrally stored in the result table to ensure that the verification process is traceable and reversible.

[0070] For example, the entry text in line 75 of the second result file contains three field segments: the input type is identified as a table-based question and answer type, and the field names are question number, table identifier, and query condition, respectively. The mapping relationship table is looked up by the current line number to obtain the assumed use case number 75. The input feature registered by number 75 is read from the use case list as a text summary type, and the field names are summary number and original paragraph, respectively. When comparing element by element, the values ​​of the input category elements are inconsistent and the number of field name elements is unequal. It is determined that the content feature does not match the registered input feature. The suspected misalignment value is registered in line 75, and the line number of line 75 and the content feature vector are stored in the temporary storage area.

[0071] Preferably, step S3, which marks entries with inconsistent content features as suspected misalignment entries, may further include the following steps: Step S3.31: For entries marked as suspected misalignment entries, compare the entry content features with the entry content features of the use case number plus one in the first result file; Step S3.32: If they match, determine that a single line misalignment has occurred and record the number of misaligned lines; Step S3.33: If it does not match, continue to compare the content features with the items in the first result file whose test case number is two or three times higher, until an item with matching content features is found or the comparison reaches the end of the preset adjacent range. When a matching item is found, record the number of misaligned lines. Step S3.34: If the content characteristics of an entry do not match the content characteristics of any adjacent entry in the first result file, the entry is determined to be an output content anomaly. A content anomaly identifier is generated, and a content anomaly review request is sent to the execution engineer terminal to complete the marking of the suspected misaligned entry.

[0072] It should be noted that the object of the misaligned line count comparison is each suspected misaligned entry registered in the temporary storage area. The comparison benchmark is the neighboring entry retrieved by offset of the use case number in the first result file. During the retrieval, the use case number to which the suspected entry is assumed to belong is used as the benchmark number. The benchmark number is incremented by one, two, and so on until it reaches the upper limit of the preset neighboring range. At each step, the offset number is used as the key to retrieve the entry position in the mapping relationship table of the first result file. Then, the entry is read from the corresponding position in the first result file and the content feature vector is extracted in the same way as in step S3.1.

[0073] Specifically, the first comparison corresponding to the base number plus one compares the content feature vector of the suspect entry with the content feature vector of the entry with the number plus one element by element. When the number of elements in both is equal and the value of each corresponding element is consistent, they are judged to be a match. At this time, it is determined that the entry has a single-line misalignment, and the misalignment line number is recorded as 1. The registration position is the misalignment line number column of the suspect entry in the temporary storage area. When the comparison does not match, it moves to the comparison of the base number plus two. The comparison method is exactly the same as the comparison of the first one. When they match, the misalignment line number is recorded as 2. If they still do not match, it continues to move to the comparison of the base number plus three. This process continues in a way that increases the number sequentially.

[0074] Furthermore, the termination condition of the advancement process is limited by a preset proximity range. When the sum of the baseline number and the offset reaches the upper limit of the preset proximity range and no entry matching the content characteristics is found, the advancement stops and the process switches to the real anomaly determination branch. The preset proximity range is obtained by statistically analyzing the maximum number of missing entries due to network interruption in a single round of execution. In this embodiment, the preset proximity range is preferably 5 number offsets. In practical applications, it can be adjusted within the range of 3 to 10 number offsets. In the joint debugging environment where the single output volume of the test case is large, a smaller value is used to control the comparison overhead, and in the environment where missing items occur frequently, a larger value is used to improve the recall coverage.

[0075] It should be noted that when all neighbor comparisons are inconsistent, the entry is determined to be a true anomaly in the output content. That is, the entry is not caused by a misaligned line number, but rather the output content of the AI ​​subsystem under this use case deviates from the output of the first round. At this time, a true anomaly identifier is generated. The identifier includes the line number of the entry, the assumed use case number, and the content feature vector. The true anomaly identifier is sent to the execution engineer's terminal through the message channel to request a content anomaly review. The review request carries the entry text so that the execution engineer can view it directly.

[0076] Once the content authenticity anomaly identifier is generated, the entry is removed from the boundary judgment candidate of the misalignment suspected entry and no longer participates in the boundary consistency judgment in steps S3.5 to S3.8. It is only presented separately as an authenticity anomaly entry in the difference report.

[0077] In practical applications, the misaligned row number field is allowed to be updated during the comparison process. When a suspected item first fails to match in the comparison with number plus one, and then matches in the comparison with number plus two, the misaligned row number field is changed from empty to 2, and the field value is based on the final offset of the match.

[0078] Furthermore, each suspected misaligned entry in the temporary storage area has one of three landing points after the comparison is completed: the number of misaligned lines is registered as 1, the number of misaligned lines is registered as 2 or a value between the upper limit of the preset adjacent range, or it is removed from the suspected candidate and a true anomaly mark is attached. The three landing points are distinguished by different mark field values ​​in the temporary storage area, which are used for boundary consistency judgment to filter and read according to the mark field.

[0079] For example, the suspected misalignment entry in row 75 of the temporary storage area, assuming it belongs to use case number 75, compares its content feature vector with that of entry number 76. The input category elements are the same, but the field name element sequence is different, so it is determined to be inconsistent. It continues to compare with the content feature vector of entry number 77. The number of elements is equal and all corresponding elements have the same value, so it is determined to be consistent, and the misalignment row number field is registered as 2. The suspected entry in row 90 of the same temporary storage area is compared with the content feature vectors of entries numbered 91 to 95 one by one and they are inconsistent. The comparison reaches the upper limit of the preset adjacent range, so it is determined to be a real anomaly in the output content. A real anomaly label is generated and sent to the execution engineer terminal, completing the marking of the suspected misalignment entry.

[0080] In this embodiment of the application, step S3, which performs boundary consistency judgment based on the missing use case list and the suspected misaligned items to obtain the misalignment boundary identifier containing the starting item position data, may further include the following steps: Step S3.5: Determine the number of the first failed use case based on the missing use case list, and determine the entry position of the first failed use case number in the first result file based on the use case number mapping table, as the candidate misalignment starting boundary; Step S3.6: If the position of the first suspected misaligned entry in the first result file is consistent with the candidate misalignment start boundary, and the number of misaligned rows of the suspected misaligned entry is consistent with the number of previously accumulated missing test cases, generate a misalignment boundary identifier containing the start entry position data and the number of misaligned rows data to complete the boundary consistency judgment. Step S3.7: If all entries in the second result file are marked as belonging to normal entries, generate a no-misalignment identifier and complete the boundary consistency judgment; Step S3.8: If a suspected misalignment item exists but the item position is inconsistent with the candidate misalignment starting boundary, or the number of misaligned rows does not match the number of missing test cases, generate a boundary uncertainty flag, send a boundary verification request to the comparison engineer terminal, and complete the boundary consistency judgment.

[0081] Understandably, the purpose of boundary consistency judgment is to confirm that the line number offset reflected by the suspected misaligned entries and the forward shift of entries caused by missing test cases are numerically consistent. Since all entries after the missing test cases are shifted forward as a whole, the position of the first misaligned entry should theoretically fall exactly on the position of the first failed test case in the first result file, and the number of misaligned lines should be equal to the cumulative number of missing test cases. Only when the two values ​​match can the misalignment boundary be considered credible.

[0082] Specifically, the candidate misalignment starting boundary is determined by reading the first number from the ordered number list of the missing use case list as the first failed use case number. Then, using this number as the search key, the corresponding first result file entry position field is searched in the use case number mapping table. The entry position value read is the candidate misalignment starting boundary, and the boundary value is registered in the boundary judgment work area in the form of a row number.

[0083] Furthermore, the first suspected misalignment entry is selected by scanning the content attribution verification result table according to the line number order of the second result file, reading the first line number registered as a suspected misalignment value that has not been removed from the suspected candidate, and then reading the corresponding entry position of the line number entry obtained during the comparison process from the temporary storage area. This entry position is then compared with the candidate misalignment starting boundary. When the two values ​​are equal, the position is determined to be consistent. When the two values ​​are not equal, the position is determined to be inconsistent and the process is transferred to the boundary uncertainty branch.

[0084] It should be noted that the determination of the consistency between the number of misaligned rows and the cumulative number of missing test cases uses a numerical equality comparison. The misaligned row count field of the first suspected misaligned entry is retrieved from the temporary storage area. The cumulative number of missing test cases is obtained by counting the number of entries in the missing test case list whose test case number is less than or equal to the number of the first failed test case. The two values ​​are then directly compared. If they are equal, the test case is considered consistent; if they are not equal, the test case is considered inconsistent, and the process proceeds to the boundary uncertainty branch. The detailed execution methods for the above two determinations include the following sub-processes:

[0085] A1. Read the first number in the ordered list of missing use cases as the first failed use case number in the list order, write the first failed use case number into the boundary judgment work area, and proceed to step A2.

[0086] A2. Using the first failed test case number as the search key, traverse the number column of the test case number mapping table. After a registration item is matched, read the value of the first result file entry position field of the registration item, write the value into the candidate boundary column of the boundary judgment work area, and proceed to step A3.

[0087] A3. Scan the content ownership verification result table in line number order, read the line number corresponding to the first suspected misalignment value, use the line number as the key to search for the corresponding entry position and misalignment line number value in the first result file of the entry in the temporary storage area, and proceed to step A4.

[0088] A4. Compare the corresponding entry position of the first result file read from the temporary storage area with the value of the candidate boundary field. If they are equal, proceed to step A5. If they are not equal, proceed to step S3.8 to generate the boundary uncertainty identifier.

[0089] A5, count the number of entries in the missing test case list whose number is less than or equal to the number of the first failed test case, record the statistical result as the cumulative number of missing test cases, read the value of the misaligned row number field in the temporary storage area, compare the value of the misaligned row number with the cumulative number of missing test cases, if the two are equal, proceed to step A6, if the two are not equal, proceed to step S3.8 to generate boundary uncertainty identifier.

[0090] A6. Take the candidate boundary field value as the starting entry position data and the misaligned row number value as the misaligned row number data, assemble and write the misaligned boundary identifier, complete the boundary consistency judgment, and proceed to the report pause process in step S4.

[0091] In this embodiment, the misalignment boundary identifier is stored as an independent record in the boundary judgment work area. The record contains four fields: the starting entry position data, the number of misaligned rows, the first failed test case number, and the generation time. These fields are available for the report pause and re-pairing process in step S4 to read as needed.

[0092] Furthermore, the determination that all entries in the second result file are marked as belonging to normal entries is completed by reading the content belonging verification counter. When the value of the counter for suspected misalignment entries is 0 and the number of true content anomaly markers is 0, all entries are determined to belong to normal, and a misalignment-free marker is generated. After the misalignment-free marker is generated, the report pause process is skipped, and the difference report is directly generated and archived according to the normal path aligned with the line number.

[0093] Furthermore, the triggering condition for the boundary uncertainty branch is that either the position inconsistency or the row number inconsistency is met. After triggering, a boundary uncertainty identifier is generated. The identifier contains four values: the candidate misalignment starting boundary, the position of the first suspected misalignment entry, the misaligned row number, and the cumulative number of missing test cases. The boundary uncertainty identifier, along with the boundary review request, is sent to the comparison engineer's terminal via a message channel. The four values ​​are presented side by side in the review request so that the comparison engineer can directly determine the source of the discrepancy. After receiving the review conclusion, the boundary judgment work area is updated according to the conclusion, and the judgment in steps A4 and A5 is re-executed.

[0094] In practical applications, the boundary judgment work area is cleared and rebuilt at the beginning of each round of comparison to avoid the boundary values ​​of the previous round remaining in the judgment of the current round. All intermediate values ​​in the work area are retained along with the task record when the comparison task is archived.

[0095] For example, the ordered list of missing use cases is 75 and 76. The first number 75 is read as the first failed use case number. The position of the entry with number 75 in the first result file is found in the mapping table as row 75, and the candidate boundary field is registered as 75. The first suspected misaligned entry is found in row 75 by scanning the result table. The corresponding entry in the first result file is found in row 77 from the temporary storage area, and the number of misaligned rows is 2. The two values ​​are compared and the entry positions 75 and 77 are not equal. The process is to switch to the boundary uncertainty branch, generate a boundary uncertainty identifier containing candidate boundary 75, suspected entry position 77, misaligned row number 2, and cumulative missing number 2, and send a boundary review request. The review conclusion confirms that the missing use cases are indeed numbered 75 and 76 and the entry in row 75 corresponds to number 77. After updating, the position comparison and row number comparison are re-executed. Both are consistent, and a misaligned boundary identifier containing the starting entry position data 75 and the misaligned row number data 2 is generated, completing the boundary consistency judgment.

[0096] Step S4: In response to the misalignment boundary marker, a report pause instruction is generated. The second result file entries are re-paired with the entries with the same use case number in the first result file according to the use case number mapping table, and a difference report is generated. The use case number mapping table records the correspondence between each use case number in the use case list and the positions of the first and second result file entries. It should be noted that the report pause command applies to the archive queue and display interface of the difference report. After the command is generated, the archive queue stops receiving new entries, and the display interface stops refreshing the difference conclusions. This prevents difference entries generated in a misaligned state from entering the archive and display chain. The pause state is lifted after the re-pairing is completed and the difference conclusions are confirmed.

[0097] Specifically, the re-pairing is based solely on the use case number mapping table. During pairing, the use case numbers are read one by one in the order of the use case list. Using the number as the key, the position of the first result file entry and the position of the second result file entry are retrieved in the mapping table. The corresponding entries are extracted from the two result files to form a pairing group. The two entries in the pairing group are compared according to their content to generate a difference conclusion. The row corresponding to the missing use case is left blank in the pairing group and marked as an entry to be supplemented.

[0098] Furthermore, the difference report is assembled in the order of use case numbers after all paired groups have completed the comparison. Each difference entry in the report carries a use case number, and the entry corresponds one-to-one with the use case number, thereby eliminating false differences caused by misaligned line numbers. The specific steps for generating and pausing the report are explained in the extended content of steps S4.1 to S4.4.

[0099] In some embodiments, step S4, in response to the misalignment boundary marker, generates a report pause instruction, and after re-pairing the entries in the second result file with the entries in the first result file with the same use case number according to the use case number mapping table to generate a difference report, may further include the following steps: Step S4.1: In response to the misalignment boundary marker, generate a report pause command to pause the archiving and display of the difference report; Step S4.2: Generate a realignment instruction. Pair the entries in the second result file with the entries in the first result file with the same use case number according to the use case number mapping table, and compare the contents. Leave the rows of missing use cases blank and mark them as entries to be supplemented. Step S4.3: Wait for the alignment completion feedback signal after pairing is completed. If no alignment completion feedback signal is received within the preset waiting time, resend the realignment command. If no realignment is received again, send a manual alignment request to the comparison engineer terminal. Step S4.4: Send a paused notification signal containing the pause reason and a summary of the missing use case list to the comparison engineer terminal and the execution engineer terminal respectively, and generate a difference report according to the re-paired items.

[0100] Understandably, the report pause instruction is generated immediately after the misalignment boundary marker is read. The instruction includes the pause object identifier and the pause reason field. The pause object identifier points to the current comparison task's archive queue number and display interface number. After receiving the instruction, the archive queue is set to pause receiving state, and after receiving the instruction, the display interface sets the difference conclusion area to pause refresh state. Both states remain unchanged until the pause is lifted.

[0101] Specifically, the realignment instruction is generated by assembling a task message containing a mapping table identifier, two result file identifiers, and a pairing start number. During pairing execution, each use case is processed sequentially from the first number in the use case list. For the current number, the position of the first result file entry and the position of the second result file entry are retrieved in the mapping table. If both positions exist, the entries are read from the two files respectively, and the differences are generated by comparing each field according to the content. Numbers where the second result file entry position does not exist are determined to be missing use cases. The row position of this number is left blank in the pairing table and marked as an entry to be supplemented. After all numbers have been processed, the pairing table is fully registered. The pairing table saves the difference conclusion or supplementation mark for each number in numerical order.

[0102] Furthermore, the waiting for the alignment completion feedback signal is executed by the listening timer of the report control module. When the pairing process ends, a feedback signal is sent to the listening side. The listening timer starts when the realignment command is issued. When a feedback signal is received within the alignment feedback time, the process switches to generating a difference report. If the timer expires and no feedback is received, the realignment command is resent and the timer is restarted. If no feedback is received after the second timer expires, a manual alignment request is sent to the comparison engineer's terminal. The request carries the current progress of the pairing table and a list of incomplete numbers. The comparison engineer confirms the pairing results of the remaining numbers on the terminal and then sends them back. The alignment feedback time is used to limit the upper limit of the response waiting time for the realignment command. It is obtained by statistically analyzing the actual execution time of the pairing process in historical comparison tasks. In this embodiment, the alignment feedback time is preferably 60 seconds. In practical applications, it can be adjusted within the range of 30 seconds to 180 seconds. The larger value is used for comparison tasks with larger result file entries, and the smaller value is used for tasks with smaller entry sizes.

[0103] It should be noted that the method for sending the paused notification signal of the difference report is to assemble a message containing a pause reason field, a summary of the missing use case list, and the start and end times of the pause. The summary of the missing use case list is composed of the first and last numbers extracted from the ordered numbered list of missing use cases and the total number of the list. The same message is pushed to the comparison engineer terminal and the execution engineer terminal respectively. The push channels of the two terminals are independent of each other, and the push results of each channel are registered separately.

[0104] In practical applications, the action of generating a difference report based on the re-paired items is executed after the pairing table is fully registered. The report is assembled line by line in the order of the use case number. Each line carries the use case number, the content of the first result file item, the content of the second result file item, and the difference conclusion. The line corresponding to the missing use case carries a mark to be filled and a blank line position. After the report is assembled, it is still in a paused state, waiting for the rerun and secondary confirmation process of steps S4.31 to S4.33 to determine whether to lift the pause or maintain the pause.

[0105] In this embodiment, during the paused state, the difference report is neither written to the archive queue nor displayed on the display interface. The temporary storage location of the report is the cache area of ​​the report control module, and the report content in the cache area is refreshed synchronously with the update of the pairing table.

[0106] For example, after reading the misalignment boundary identifier, a report pause command is generated, the archiving queue is set to pause receiving state, and the display interface pauses refreshing; the realignment command is paired one by one according to the numbers 1 to 200. The entries with numbers 75 and 76 do not exist in the second result file. The row positions of the two numbers are left blank and marked as entries to be supplemented. The remaining numbers are paired to generate the difference conclusion; after the realignment command is issued, an alignment completion feedback signal is received after 45 seconds, and the process switches to difference report generation. After the report is assembled in numerical order, it is temporarily stored in the cache area. At the same time, a difference report pause prompt signal containing the pause reason and the summary of the missing use case list with numbers 75 to 76 is pushed to the comparison engineer terminal and the execution engineer terminal respectively.

[0107] In some preferred embodiments, step S4, in response to the misalignment boundary marker, generates a report pause instruction, and after re-pairing the entries in the second result file with the entries in the first result file with the same use case number according to the use case number mapping table to generate a difference report, may further include the following steps: Step S4.31: In response to the alignment completion feedback signal, generate a missing test case rerun query and send it to the execution engineer terminal. The query contains a list of missing test cases and two options: rerun the missing test cases or generate a difference conclusion based on the existing items. Step S4.32: If the execution engineer chooses to rerun the missing test cases, the missing test case list is converted into a rerun task. After the rerun test cases are completed, an execution status identifier is generated and written to the record table. The rerun results are appended to the corresponding line position in the second result file. If the engineer chooses to generate the difference conclusion based on the existing items, the missing test cases are marked as uncompared in the difference report. Step S4.33: After the rerun is completed, perform a second confirmation judgment: If the items are complete after the rerun and all the previously suspected misaligned items are converted to consistency after being paired according to the use case number, generate a false difference confirmation mark and release the report from the pause state; if the items are complete after the rerun but there are still items whose content does not match the items with the same use case number in the first result file, generate a real difference mark; if the rerun fails again, or the rerun result does not match the content characteristics of the adjacent items in either the first result file or the second result file, generate a question mark, send a manual review request to the team leader's terminal, maintain the report from the pause state, and generate a difference report according to the re-paired items.

[0108] It should be noted that the missing test case replacement query is assembled immediately after the alignment completion feedback signal arrives. The query content includes an ordered list of missing test cases and two option button labels, which are pushed to the execution engineer terminal via the message channel. After the push, the query response timer is started, and the response direction is determined by the value of the received button message.

[0109] Specifically, the method for issuing supplementary test cases is as follows: each number in the missing test case list is restored to the complete input content of the test case, and then dispatched to the AI ​​subsystem for execution in numerical order. When each supplementary test case is executed, an execution status identifier is generated and written to the record table in the same way as in step S1.1. The round field in the identifier is filled with the supplementary test case marker. The supplementary test case result is returned in the form of a result entry. The append action uses the test case number of the supplementary test case as the key to retrieve the corresponding row position in the second result file in the test case number mapping relationship table, writes the supplementary test case result entry to that row position, and updates the second result file entry position field of that number in the mapping relationship table after writing. When the difference conclusion is generated according to the existing entries, no dispatch action is performed on the missing test cases. Instead, the non-comparison status is directly registered for each row position corresponding to the number in the missing test case list in the difference report in the cache area. After the registration is completed, the report remains in a paused state, waiting for archiving confirmation.

[0110] Furthermore, the secondary confirmation judgment is performed after all the rerun tasks are completed or the non-comparison status registration is completed. The input of the judgment includes the completeness status of the entries in the second result file after the rerun and the pairing result of the misaligned suspected entries. The completeness status of the entries is obtained by comparing the current total number of entries in the second result file with the total number of use cases in the use case list. The pairing result is obtained by re-pairing and comparing each misaligned suspected entry in the temporary storage area according to the real use case number corresponding to its content characteristics.

[0111] It should be noted that the determination that all items are complete and all suspected misaligned items are converted to consistency is based on two conditions: the total number of items equals the total number of test cases, and the content feature vector of each suspected misaligned item in the temporary storage area is consistent element-by-element with the item with the same number in the first result file after being paired by test case number. When both conditions are met, a false difference confirmation mark is generated, confirming that the previous difference items were all caused by line number misalignment. After the false difference confirmation mark is generated, the report pause status is lifted, and the difference report in the cache is written to the archive queue in the lifted state and presented on the display interface; if all items are complete but there are still items that do not match the item with the same number in the first result file, then... Generating a true difference identifier, and marking discrepancies separately in the report as true differences; the rerun fails when the execution result status data is registered as a failure value at the end of the rerun test case, or when the content feature vector of the rerun result item does not match the content feature vector of the same numbered item in the first result file and any adjacent item in the second result file element by element. When either of these conditions is met, a question mark is generated, and the question mark along with a manual review request is sent to the team leader's terminal via the message channel. The report remains in a paused state, and the report is generated according to the re-paired items and temporarily stored in the cache to await the review conclusion.

[0112] The manual review request is obtained by automatically assembling a request message when the question mark is generated. The message content includes the test case number of the questioned item, the rerun execution result, the content feature comparison result, and the start and end times of the pause. In this embodiment, it is preferred that the test platform automatically assembles and pushes the message. In practical applications, a summary of the interaction records between the comparison engineer and the execution engineer can be added to the request message so that the team leader can trace the entire processing process.

[0113] In practical applications, the three types of markers—false difference confirmation marker, true difference marker, and questionable marker—are mutually exclusive in the boundary judgment work area. Once any marker is generated, the secondary confirmation judgment ends immediately, thus avoiding the coexistence of conclusions in the same comparison task.

[0114] Meanwhile, the actions to lift the report's pause status include restoring the archiving queue to the receiving state, restoring the display interface to the refresh state, and writing the start and end times of the pause to the interaction log table. After these three actions are executed in sequence, the report enters the normal archiving and display chain.

[0115] For example, after the alignment completion feedback signal arrives, the supplementary run query carrying numbers 75 and 76 is pushed to the execution engineer's terminal. The execution engineer presses the supplementary run button, and after the two supplementary run test cases are dispatched and executed, they both return result entries. Each generates an execution status identifier, writes it into the record table, and appends it to lines 75 and 76 of the second result file. In the secondary confirmation judgment, the total number of entries 200 equals the total number of test cases 200. The misaligned suspected entries in line 75 in the temporary storage area are re-paired according to the real test case number 77, and the content characteristics are consistent element by element. The real abnormal entries in line 90 are marked separately according to the real difference, a false difference confirmation identifier is generated, the report pause status is lifted, the difference report is written to the archive queue and presented on the display interface.

[0116] Figure 2 This section illustrates a segment of the data acquisition and integrity assessment process. The process begins with acquiring the execution status identifier and two result files. Then, network interruption records are introduced to perform segmented verification, thereby eliminating interfering data. After data cleaning, the number of successful entries is compared with the total number of test cases. If they are equal, the missing case list generation step is skipped, and subsequent verification proceeds directly. If the number is less than the total number of test cases, failure identifiers are extracted to generate a missing test case list, and an integrity warning signal is sent to the terminal. At this point, the task's direction is determined by the button response: selecting "pause" suspends the task and ends the current process; selecting "continue" or "no response after timeout" binds the list and proceeds to the feature extraction stage on the next page.

[0117] Figure 3This page illustrates a segment of the content attribution verification and proximity comparison process. Following the previous branch, it iterates through the second result file to extract entry content features and matches them against the registered input features. Matching entries are marked as having normal attribution, while mismatched entries are marked as potentially misplaced and proceed to the proximity comparison stage. During proximity comparison, if no matching entry is found within a preset range, it is determined to be a genuine anomaly, a review request is sent, and the process ends. If a matching entry is found, the specific number of misplaced lines is recorded, providing data support for subsequent boundary consistency judgments.

[0118] Figure 4 This section illustrates a segment of the boundary consistency assessment process. This page aggregates suspected entries that are classified as normal and those with misaligned row counts for joint verification. If all entries are classified as normal, a no-misalignment flag is generated, and a difference report is generated normally according to row number, then the process ends. If suspected entries exist, the process checks whether the suspected location matches the candidate boundary and whether the number of misaligned rows matches. If the checks do not match, a boundary uncertainty flag is generated, a review request is sent, and the process ends; if the checks match, a misalignment boundary flag containing the starting position and the number of misaligned rows is generated, triggering a report pause mechanism.

[0119] Figure 5 This demonstrates a segment of the report pause and realignment process. Upon receiving a misalignment boundary marker, a report pause command is generated to prevent the archiving and display of error difference reports. Subsequently, the entries in the two result files are re-paired and compared according to the mapping table, while continuously listening for alignment completion feedback signals. If no feedback is received within a preset time, the alignment command is resent or manual intervention is requested to terminate the current automated process; if feedback is successfully received, preparations are made to proceed to the missing use case rerun query stage.

[0120] Figure 6 This section illustrates a segment of the process for requesting and confirming a supplementary test. Based on the alignment completion feedback, a supplementary test request is sent to the execution engineer. If the request is refused, the missing test cases are marked as unmatched, a report is generated, the pause is lifted, and the process ends. If the request is approved, the supplementary test task is executed, and the results are appended to the file. A second confirmation is then performed: if all items are complete after the supplementary test and all suspected items match after pairing, a false difference confirmation flag is generated, the pause is lifted, and the process ends; if discrepancies still exist or execution fails, a true difference or questionable flag is generated, the report remains paused, and the process ends, awaiting further manual review. All branches achieve logical closure on this page.

[0121] Example 2: This example also provides an AI system integration and verification system, including: an execution status recording module, an integrity judgment module, a content ownership verification module, a boundary judgment module, and a report control module; The execution status recording module is used to generate an execution status identifier, which includes test case number data, execution result status data, and round identifier data, and write it into the record table when each test case of the AI ​​subsystem is executed. The integrity judgment module is used to respond to the comparison command and generate a missing use case list based on the execution status identifiers with execution result status of failure when the number of entries with successful execution result status in the current execution status identifier is less than the total number of use cases in the use case list. The content attribution verification module is used to perform content attribution verification on each entry in the second result file, matching the content features of the entry with the input features corresponding to the test case number mapped to the current line number of the entry, and marking entries with inconsistent content features as suspected misaligned entries. The boundary judgment module is used to perform boundary consistency judgment based on the missing use case list and the suspected misalignment item to obtain the misalignment boundary identifier containing the starting item position data. The report control module is used to generate a report pause instruction in response to the misalignment boundary identifier, and then re-pair the entries in the second result file with the entries with the same use case number in the first result file according to the use case number mapping table to generate a difference report.

[0122] The above-mentioned unit modules can be embedded in the processor of the electronic device in hardware form or independent of it, or they can be stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0124] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An AI system integration and verification method, characterized in that, include: Obtain the execution status identifier generated when the AI ​​subsystem executes test cases in rounds, as well as the first result file exported after executing all test cases in the first round of test case list and the second result file exported after executing test cases in the second round; The execution status identifier includes test case number data, execution result status data, and round identifier data; In response to the comparison command, when the number of entries with a successful execution result in the execution status identifier corresponding to the current round identifier data is less than the total number of use cases in the use case list, use case number data is extracted based on the execution status identifier with a failed execution result, and a list of missing use cases is generated. For each entry in the second result file, a content attribution check is performed. The content features of the entry are matched with the input features corresponding to the test case number mapped to the current line number of the entry. Entries with inconsistent content features are marked as suspected misaligned entries. Boundary consistency judgment is performed based on the missing test case list and the suspected misaligned entries to obtain the misaligned boundary identifier containing the starting entry position data. In response to the misalignment boundary marker, a report pause command is generated. The second result file entries are re-paired with the entries with the same use case number in the first result file according to the use case number mapping table, and a difference report is generated. The use case number mapping table records the correspondence between each use case number in the use case list and the positions of the first and second result file entries.

2. The AI ​​system integration and verification method according to claim 1, characterized in that: The execution status identifiers generated when the AI ​​subsystem executes test cases in rounds include: At the end of each test case execution in the AI ​​subsystem, an execution status identifier is generated and written to the record table. The execution status identifier also includes execution time data, interruption association identifier data, and confirmation status data. The interruption association identifier data indicates whether a network interruption occurred during the execution of the test case. A network interruption is the state in which no result is received from the AI ​​subsystem within a preset waiting time after the execution request is sent. For test cases whose execution result status is "execution failed", an execution status identifier is also generated and written to the record table when the execution ends; The execution status identifiers generated in each round are collected into a record table according to the round identifier data to obtain the execution status identifiers generated when the AI ​​subsystem executes test cases in rounds.

3. The AI ​​system integration and verification method according to claim 1, characterized in that: When the number of entries with a successful execution result in the execution status identifier corresponding to the current round identifier data is less than the total number of use cases in the use case list, use case number data is extracted based on the execution status identifier with a failed execution result, and a missing use case list is generated, including: Determine whether the number of entries with a successful execution result among all execution status identifiers corresponding to the current round identifier data is equal to the total number of use cases in the use case list; If the number is less than the total number of test cases, extract the test case number data based on the execution status identifier of the execution result status being failed, and generate a list of missing test cases by arranging them in ascending order of test case number. Send an integrity alert signal containing the test case numbers from the missing test case list to the comparison engineer's terminal; If the comparison engineer selects to continue the comparison via the confirmation button on the terminal, the list of missing test cases will be bound to the current comparison task; if the engineer selects to pause the comparison, the comparison task will be suspended and the list of missing test cases will be retained; if no button response is received within the preset waiting time, the comparison will continue and an interaction record indicating that no integrity confirmation has been received will be generated, thus completing the generation of the list of missing test cases.

4. The AI ​​system integration and verification method according to claim 1, characterized in that: The step of performing content attribution verification on each entry in the second result file involves matching the entry's content features with the input features corresponding to the test case number mapped to the current line number of the entry, and marking entries with mismatched content features as suspected misalignment entries, including: For each entry in the second result file, extract the content features that reflect the input category and data fields in the entry content; Match the entry content features with the input features registered in the use case list for the use case number mapped to the current line number of the entry; If the content features of an entry do not match the registered input features, the entry will be marked as a suspected misalignment entry. If the content characteristics of an entry match the registered input characteristics, the entry is marked as a normal entry, and the content ownership verification of each entry in the second result file is completed.

5. The AI ​​system integration and verification method according to claim 4, characterized in that: The step of marking entries with inconsistent content features as suspected misalignment entries includes: For entries marked as suspected misalignments, the entry content features are compared with the entry content features of the use case number plus one in the first result file; If they match, determine that a single line is misaligned and record the number of misaligned lines; If they do not match, continue to compare the content features with the entries in the first result file whose test case number is two or three, until an entry with matching content features is found or the comparison reaches the end of the preset adjacent range. When a matching entry is found, record the number of misaligned lines. If the content characteristics of an entry do not match the content characteristics of any adjacent entry in the first result file, the entry is determined to be an output content anomaly, a content anomaly identifier is generated, a content anomaly review request is sent to the execution engineer terminal, and the mark of the suspected misaligned entry is completed.

6. The AI ​​system integration and verification method according to claim 5, characterized in that: The step of performing boundary consistency judgment based on the missing use case list and suspected misaligned items to obtain the misalignment boundary identifier containing the starting item position data includes: The number of the first failed use case is determined based on the missing use case list, and the position of the first failed use case number in the first result file is determined based on the use case number mapping table, which serves as the candidate misalignment starting boundary. If the position of the first suspected misaligned entry in the first result file is consistent with the candidate misalignment start boundary, and the number of misaligned rows of the suspected misaligned entry is consistent with the number of previously accumulated missing test cases, a misalignment boundary identifier containing the start entry position data and the number of misaligned rows is generated to complete the boundary consistency judgment. If all entries in the second result file are marked as belonging to normal entries, a no-misalignment flag is generated, and the boundary consistency judgment is completed. If a suspected misaligned entry exists but its position is inconsistent with the candidate misalignment starting boundary, or the number of misaligned rows does not match the number of missing test cases, a boundary uncertainty flag is generated, and a boundary verification request is sent to the comparison engineer's terminal to complete the boundary consistency judgment.

7. The AI ​​system integration and verification method according to claim 1, characterized in that: In response to the misalignment boundary marker, a report pause instruction is generated. Then, based on the use case number mapping table, the entries in the second result file are re-paired with the entries in the first result file with the same use case number, and a difference report is generated, including: In response to the misalignment boundary marker, a report pause command is generated to pause the archiving and display of the discrepancy report; Generate a realignment instruction, match the entries in the second result file with the entries in the first result file with the same use case number according to the use case number mapping table, compare the contents, leave the rows of missing use cases blank and mark them as entries to be filled; Wait for the alignment completion feedback signal after pairing is completed. If no alignment completion feedback signal is received within the preset waiting time, resend the realignment command. If no realignment is received again, send a manual alignment request to the comparison engineer terminal. Send a paused notification signal containing the pause reason and a summary of the missing use case list to both the comparison engineer terminal and the execution engineer terminal, and generate a difference report based on the re-paired items.

8. The AI ​​system integration and verification method according to claim 7, characterized in that: The method further includes: In response to the alignment completion feedback signal, a missing test case rerun query is generated and sent to the execution engineer terminal. The content includes a list of missing test cases and two options: rerun the missing test cases or generate a difference conclusion based on the existing items. If the execution engineer chooses to rerun the missing test cases, the list of missing test cases will be converted into a rerun task. After the rerun test cases are completed, an execution status identifier will be generated and written to the record table, and the rerun results will be appended to the corresponding line in the second result file. If the engineer chooses to generate a difference conclusion based on the existing items, the missing test cases will be marked as uncompared in the difference report. After the rerun is completed, a second confirmation judgment is performed: if the items are complete after the rerun and all previously suspected misaligned items are made consistent after being paired by use case number, a false difference confirmation mark is generated and the report is released from suspension; if the items are complete after the rerun but there are still items whose content does not match the items with the same use case number in the first result file, a real difference mark is generated; if the rerun fails again, or the rerun result does not match the content characteristics of the adjacent items in either the first or second result file, a question mark is generated, a manual review request is sent to the team leader's terminal, the report is kept in suspension, and a difference report is generated according to the re-paired items.

9. The AI ​​system integration and verification method according to claim 1, characterized in that: The method further includes: Obtain network interruption event records, which are generated when a request wait timeout state with the AI ​​subsystem is detected, and include interruption start time data and interruption end time data; The execution status identifier is segmented and verified based on the interruption period defined by the interruption start time data and the interruption end time data. The execution status identifier generated within the interruption period is not used as the basis for judging the final execution result of the test case. Instead, the test case completion receipt returned by the AI ​​subsystem is used to determine the execution success. The test case completion receipt carries the test case number and the position of the result item. Test cases for which no completion receipt has been received and whose execution request time falls within the interruption period are registered as execution failures and added to the missing test case list. If two execution status indicators overlap for the same test case, the one with the earlier completion receipt arrival time shall be used, and the other shall be marked as a duplicate record and removed. When network interruption event logs are unavailable, a query prompt is sent to the execution engineer terminal to ask whether a network interruption has occurred and the duration of the interruption. Based on the key press response, alternative interruption duration data is generated to complete the segmented verification of the execution status identifier.

10. An AI system integration and verification system, characterized in that, It includes an execution status recording module, an integrity judgment module, a content ownership verification module, a boundary judgment module, and a report control module; The execution status recording module is used to generate an execution status identifier, which includes test case number data, execution result status data, and round identifier data, and write it into the record table when each test case of the AI ​​subsystem is executed. The integrity judgment module is used to respond to the comparison command and generate a missing use case list based on the execution status identifiers with execution result status of failure when the number of entries with successful execution result status in the current execution status identifier is less than the total number of use cases in the use case list. The content attribution verification module is used to perform content attribution verification on each entry in the second result file, matching the content features of the entry with the input features corresponding to the test case number mapped to the current line number of the entry, and marking entries with inconsistent content features as suspected misaligned entries. The boundary judgment module is used to perform boundary consistency judgment based on the missing use case list and the suspected misalignment item to obtain the misalignment boundary identifier containing the starting item position data. The report control module is used to generate a report pause instruction in response to the misalignment boundary identifier, and then re-pair the entries in the second result file with the entries with the same use case number in the first result file according to the use case number mapping table to generate a difference report.

Images