Test result evaluation method and apparatus

Abstract

The application provides a test result evaluation method and device, and relates to the field of data processing. The method obtains test data in multiple test business systems; based on a preset test identification field, the test data in each test business system is associated and matched to obtain multiple-source test data corresponding to each test identification; for the multiple-source test data corresponding to each test identification, multi-dimensional independent judgment is performed according to a plurality of preset evaluation dimensions to obtain initial validity results corresponding to each evaluation dimension; the initial validity results corresponding to each evaluation dimension are corrected by using a preset cross-dimension consistency check to obtain test evaluation results corresponding to each test identification, thereby improving test result evaluation efficiency, avoiding information omission and data error matching, and also improving consistency of evaluation results.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for evaluating test results. Background Technology

[0002] Testing and verification are essential steps in evaluating whether a product meets design requirements. Taking a vehicle's steering system as an example, it typically requires various types of testing, including durability testing, environmental testing, and performance testing, to verify whether its reliability, durability, and functionality meet design expectations. Testing personnel evaluate the validity of the test results based on the data generated during the testing process, and this evaluation directly affects whether the product can proceed to the next development stage or enter mass production.

[0003] In this process, relevant data from the entire testing process is typically stored across multiple independent information systems. Taking the steering system as an example, the testing tasks for the steering system are created and scheduled in the Test Management System (TMS); the raw test data and final test reports generated during the test are stored in the Test Data Management System (TDMS); the test benches used in the test and their calibration status are recorded in the Equipment Management System (EMS); the design version and engineering change information of the product under test are maintained in the Product Lifecycle Management System (PLM); and defects discovered during the test or quality issues triggered by test conclusions are recorded in the Issue Tracking System (ITS). When evaluating the validity of the test results, testing personnel need to manually log into each system to query the corresponding data one by one, and then manually summarize and verify it before making a judgment. This leads to low efficiency and is prone to problems such as information omissions and data matching errors, making it difficult to guarantee the accuracy of the evaluation results.

[0004] Furthermore, different researchers may have different standards for judging the validity of experimental results, and may draw different conclusions about the validity of the same experimental data, which may lead to insufficient consistency in the evaluation results. Summary of the Invention

[0005] This application provides a method and apparatus for evaluating test results, in order to solve the problems of low efficiency, easy information omission, and insufficient consistency of evaluation results in the process of evaluating test results.

[0006] In a first aspect, embodiments of this application provide a method for evaluating test results, including: Acquire test data from multiple test operation systems; Based on the preset test identifier field, the test data in each test business system are correlated and matched to obtain the multi-source test data corresponding to each test identifier; For the multi-source experimental data corresponding to each experimental identifier, multi-dimensional independent judgment is performed according to multiple preset evaluation dimensions to obtain the initial validity results corresponding to each evaluation dimension; By using a preset cross-dimensional consistency check, the initial validity results corresponding to each evaluation dimension are corrected to obtain the test evaluation results corresponding to each test identifier.

[0007] Based on the aforementioned technical content, this embodiment uses test identifiers to associate and match test data scattered across various test business systems. This breaks down data barriers between these systems, enabling rapid aggregation and accurate association of dispersed data. This overcomes the problems of low efficiency, easy omissions, and data matching errors associated with manual cross-system queries. Furthermore, by independently judging the multi-source test data corresponding to each test identifier according to multiple preset evaluation dimensions, test quality can be assessed from different perspectives, comprehensively covering various quality issues that may arise during the test and avoiding the bias that can result from single-dimensional judgments. Moreover, by using preset cross-dimensional consistency checks to correct the initial validity results of each evaluation dimension, logical conflicts and anomalies that cannot be identified when each dimension is judged individually can be discovered. The initial validity results are then adjusted in a coordinated manner to eliminate logical contradictions between dimensions, thereby significantly improving the accuracy and consistency of the test evaluation results, making the final test evaluation results more comprehensive and reliable.

[0008] In one possible implementation, the step of associating and matching test data from each test service system based on a preset test identifier field to obtain multi-source test data corresponding to each test identifier includes: For every two trial business systems, determine whether they both contain a trial identifier field; If at least one of the two test business systems does not contain a test identifier field, then for the test business system that does not contain a test identifier field, the test data creation time and test data description in that test business system are obtained. Based on the test data creation time and the test data description, the test identifier is determined, and it is determined whether the test identifiers in the two test business systems correspond. If they correspond, the test data in the two test business systems are associated according to the corresponding test identifiers in the two test business systems. Based on the test data associated with each test business system, multi-source test data corresponding to each test identifier is obtained.

[0009] This embodiment uses the creation time of the test data to match the actual execution time of the test, and combines the semantic clues in the text of the test data description to match the corresponding test identifier for the test business system that is missing the test identifier field. This overcomes the problem of data mismatch caused by the lack of associated fields in some test business systems, improves the coverage and accuracy of cross-system data association, provides a complete data foundation for subsequent multi-dimensional evaluation, and ensures the comprehensiveness and reliability of the test result evaluation.

[0010] In one possible implementation, after determining whether all tests include a test identifier field, the following is also included: If both test business systems contain a test identifier field, then determine whether the encoding format of the test identifier field in the two test business systems is the same; If the encoding format of the test identifier field in the two test business systems is different, the string similarity between the test identifiers in the two test business systems is used to determine whether the test identifiers in the two test business systems correspond. If they correspond, the test data in the two test business systems are associated according to the corresponding test identifiers in the two test business systems. If the encoding format of the test identifier field in two test business systems is the same, then the test data with the same test identifier in the two test business systems will be associated. Based on the test data associated with each test business system, multi-source test data corresponding to each test identifier is obtained.

[0011] Here, in response to the different encoding formats of the test identifier field in different test business systems, automatic fuzzy matching between test data with different encoding formats but actually pointing to the same test task is achieved by calculating the string similarity between test identifiers. This overcomes the defect that cross-system data cannot be associated through equivalent matching due to inconsistent encoding rules.

[0012] In one possible implementation, the test operation system includes a test management system; the test management system includes a test identifier field; the test data in the test management system includes the actual test execution time corresponding to each test identifier; The process of determining the test identifier based on the test data creation time and the test data description includes: Based on the actual execution time of each test identifier in the test management system, calculate the time difference between the test data creation time and the actual execution time of each test identifier. If the time difference is less than the set time difference, then the test data of the test business system that does not contain the test identifier field will be determined as the candidate matching item for the corresponding test identifier. For each candidate matching item corresponding to each test identifier, a regular expression scan is performed on the test data description in the candidate matching item. If the regular expression scan result matches the test identifier, then the test identifier is determined as the test identifier of the candidate matching item.

[0013] Here, a two-stage matching strategy combining time filtering and regular expression scanning is adopted to match corresponding test identifiers for test business systems that lack a test identifier field. Time filtering utilizes the temporal relationship between the creation time and actual execution time of the test data to quickly narrow down the candidate matching range; regular expression scanning further extracts implicit identifier information from the test data description to accurately confirm the matching relationship, thereby solving the problem of data not being automatically associated due to the lack of a test identifier field in some test business systems.

[0014] In one possible implementation, the initial validity results corresponding to each evaluation dimension include: the judgment result and the basic score corresponding to each evaluation dimension; The process of using a preset cross-dimensional consistency check to correct the initial validity results corresponding to each evaluation dimension, and obtaining the test evaluation results corresponding to each test identifier, includes: Based on the logical constraints between the various evaluation dimensions and the judgment results corresponding to each evaluation dimension, the basic scores corresponding to each evaluation dimension are corrected to obtain the target scores corresponding to each evaluation dimension. Obtain the test type corresponding to each test identifier, and determine the weight of each evaluation dimension in each test identifier based on the test type; Based on the weights and the target scores corresponding to each evaluation dimension in each test identifier, the comprehensive score of each test identifier is determined. Based on the comprehensive score of each test label, the test evaluation result corresponding to each test label is determined.

[0015] This application's embodiments utilize the logical constraints between various evaluation dimensions to perform cross-dimensional consistency verification of initial validity results. This enables the identification of logical conflicts that would be undetectable when each dimension is judged individually, eliminating causal contradictions between dimensions and improving the accuracy and consistency of evaluation results. Simultaneously, differentiated weights are assigned to each evaluation dimension based on the characteristics of different experiment types, allowing the evaluation system to adapt to the focus of different experiment types and improving the relevance and rationality of experiment evaluation. Furthermore, the corrected results for each dimension are quantified into a comprehensive score, and the experiment evaluation result is determined based on this comprehensive score. This provides a unified quantitative standard for the evaluation conclusions, avoiding subjective judgment differences among different experimenters and achieving an objective and standardized evaluation of the validity of experiment results.

[0016] In one possible implementation, the test type includes durability testing; If the test type corresponding to each test label is a durability test, the determination of the test evaluation result corresponding to each test label based on the comprehensive score of each test label includes: The test identifiers are aggregated to obtain a test chain; the test chain contains multiple test identifiers arranged in sequence. For each test chain, based on the order of the test identifiers in the test chain and the comprehensive score corresponding to each test identifier in the test chain, the penalty deduction value corresponding to each test identifier in the test chain is determined; Based on the comprehensive score of each test identifier and the penalty deduction value, the corresponding test evaluation result for each test identifier is determined.

[0017] This embodiment addresses multi-stage continuous testing, such as durability testing. It aggregates multiple test stages belonging to the same product under test and the same testing equipment into a test chain, and applies penalty points based on the chronological order of the test identifiers within the chain. This ensures that the failure state of a preceding test can be propagated along the chain. Consequently, it guarantees causal relationships and logical consistency between the evaluation results of each stage within the same test chain, improving the reliability and comprehensiveness of multi-stage test evaluation results.

[0018] In one possible implementation, the aggregation of each test identifier to obtain a test chain includes: According to the product code under test in the multi-source test data corresponding to each test identifier, the test identifiers are grouped; each group contains multiple test identifiers, and the multiple test identifiers correspond to the same product code under test. For each group, sort them in ascending order according to the start time of the test plan corresponding to each test identifier to obtain the test sequence corresponding to that group; Traverse the test sequence. If the test time interval between two adjacent test identifiers is less than the set time interval, and the test equipment corresponding to the two adjacent test identifiers is the same, then it is determined that the two adjacent test identifiers satisfy the aggregation relationship. Multiple test identifiers that consecutively satisfy the aggregation relationship in the test sequence are aggregated into the same test chain.

[0019] In this embodiment, information from three dimensions—product code, test plan start time, and test equipment—is used to achieve automatic identification and aggregation of each test stage in durability testing. This provides a structural basis for the propagation of preceding failures to subsequent failures, improving the accuracy and reliability of multi-stage test evaluation results.

[0020] In one possible implementation, for each test chain, based on the order of the test identifiers in the test chain and the comprehensive score corresponding to each test identifier in the test chain, the penalty deduction value corresponding to each test identifier in the test chain is determined, including: For each test identifier in each test chain, check whether the overall score of the preceding test identifier is lower than the set score; If the overall score of the preceding test identifier is lower than the set score, then the difference between the overall score of the preceding test identifier and the set score is calculated. Based on the difference, the penalty score corresponding to the test identifier is determined; the higher the difference, the higher the penalty score.

[0021] This embodiment establishes a quantitative penalty mechanism for the transmission of failure status of preceding tests to subsequent tests within a test chain, enabling the failure risk of preceding tests to be quantitatively transferred to subsequent tests. This effectively prevents situations where subsequent tests are still deemed valid despite the failure of preceding tests, ensuring causal consistency and logical coherence among the evaluation results of each stage of the same test chain, and improving the reliability and comprehensiveness of multi-stage test evaluation results.

[0022] In one possible implementation, the preset multiple evaluation dimensions include: test data dimension, test equipment dimension, tested product dimension, and test index dimension; the judgment result includes: pass or fail. The process involves correcting the base scores for each evaluation dimension based on the logical constraints between the various evaluation dimensions and the judgment results corresponding to each evaluation dimension, thereby obtaining the target scores for each evaluation dimension. This includes: If the test equipment dimension fails the test but the test data dimension passes the test, the base score of the test data dimension is reduced to obtain the target score of the test data dimension. If the tested product dimension fails the test, the base score of the test indicator dimension is reduced to obtain the target score of the test indicator dimension.

[0023] Here, the basic scores in the initial validity results are corrected by using the logical constraints between the evaluation dimensions, eliminating the logical contradictions that may arise when each dimension is judged individually. This ensures that the final target score can accurately reflect the impact of the logical constraints between the dimensions on the evaluation conclusion, thereby improving the accuracy and logical consistency of the evaluation results.

[0024] Secondly, embodiments of this application provide a test result evaluation device, comprising: The acquisition module is used to acquire test data from multiple test business systems.

[0025] The matching module is used to perform correlation matching on test data in each test business system based on a preset test identifier field, so as to obtain multi-source test data corresponding to each test identifier.

[0026] The evaluation module is used for: For the multi-source experimental data corresponding to each experimental identifier, multi-dimensional independent judgment is performed according to multiple preset evaluation dimensions to obtain the initial validity results corresponding to each evaluation dimension; By using a pre-defined cross-dimensional consistency check, the initial validity results corresponding to each evaluation dimension are corrected to obtain the test evaluation results corresponding to each test identifier.

[0027] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the test result evaluation method as described in any of the first aspects.

[0028] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the test result evaluation method as described in any of the first aspects.

[0029] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application; Figure 2 This is a schematic flowchart of an embodiment of the test result evaluation method provided in this application; Figure 3 This is a flowchart illustrating the process of associating and matching test data in various test service systems, provided in one embodiment of this application. Figure 4 This is a flowchart illustrating a test result evaluation method provided in another embodiment of this application; Figure 5 This is a schematic diagram of the structure of an experimental result evaluation device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The present application will be described more clearly below with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the function of the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application. These all fall within the protection scope of the present application.

[0034] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0035] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0036] In the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0038] Furthermore, the term "multiple" mentioned in the embodiments of this application should be interpreted as two or more.

[0039] The applicant found that, in order to accurately evaluate the test results of the tested products, testers typically need to query corresponding data from multiple independent information systems one by one, and then summarize and verify the data to obtain the evaluation results. Taking the steering system in a vehicle as an example, various types of tests are usually required, such as durability tests, environmental tests, and performance tests. In the entire testing process, the creation and scheduling of test tasks are managed by the TMS system; the original test data and the final test report are stored in the TDMS; the test equipment used in the test process (i.e., the test bench) and its calibration status are recorded in the EMS; the design version and engineering change information of the tested product (i.e., the steering system) are maintained in the PLM; and test defects and quality problems are recorded in the ITS. When evaluating the validity of the test results, testers need to manually log into each system to query the corresponding data one by one, and then manually summarize and verify the data to make a judgment. The entire test evaluation process is prone to problems such as low efficiency, information omissions, and data matching errors. In addition, different testers have different standards for controlling the validity of test results, and different testers may draw different validity conclusions for the same test data, resulting in insufficient consistency of evaluation results. Therefore, it is necessary to consider a new method to evaluate the validity of the experimental results.

[0040] To improve the efficiency of test evaluation, avoid information omissions and data matching errors, and enhance the consistency of evaluation results, this application's implementation uses test identifiers to associate and match test data scattered across various test business systems. This breaks down data barriers between these systems, enabling rapid aggregation and accurate association of dispersed data, thus overcoming the problems of low efficiency, easy omissions, and data matching errors associated with manual cross-system queries. Furthermore, by independently judging the multi-source test data corresponding to each test identifier according to multiple preset evaluation dimensions, and correcting the initial validity results using preset consistency checks, the potential bias caused by single-dimensional judgments can be avoided, logical contradictions between dimensions can be eliminated, and the accuracy and consistency of test evaluation results can be significantly improved.

[0041] First refer to Figure 1 , Figure 1 A schematic diagram of an application scenario provided according to an embodiment of this application is shown, in which the device involved in the application scenario includes an electronic device 101.

[0042] When evaluating test results, the electronic device 101 can acquire test data from multiple test business systems and perform correlation matching on the test data in each test business system based on a preset test identifier field to obtain multi-source test data corresponding to each test identifier. Then, for the multi-source test data corresponding to each test identifier, it performs multi-dimensional independent judgment according to multiple preset evaluation dimensions to obtain the initial validity results corresponding to each evaluation dimension. Finally, it uses a preset cross-dimensional consistency check to correct the initial validity results corresponding to each evaluation dimension to obtain the test evaluation results corresponding to each test identifier, thereby realizing the validity evaluation of the test results.

[0043] Optionally, the application scenario also includes storage devices 102 running the aforementioned test service systems. Electronic devices 101 and storage devices 102 can communicate via a network to acquire test data from each test service system. It is understood that the multiple test service systems can run on different storage devices or on the same storage device. The electronic devices are communicatively connected to each storage device running a test service system. Figure 1 Only one storage device is shown as an example.

[0044] The following is combined with Figure 1 Application scenarios, refer to Figures 2-4 This application describes a method for evaluating test results based on exemplary embodiments thereof. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way. Rather, the embodiments of this application can be applied to any applicable scenario.

[0045] refer to Figure 2 , Figure 2 This is a schematic flowchart illustrating a test result evaluation method provided in one embodiment of this application. Figure 2 As shown, the method in the embodiments of this application may include: Step 201: Obtain test data from multiple test business systems.

[0046] This embodiment can evaluate test results from different dimensions, including test task information, raw test data, calibration status of test equipment used during the test, engineering change information of the tested product, and defect records during the test. These data are typically stored in different test operation systems. This embodiment can use a standard service interface to retrieve the data from each test operation system for test result evaluation.

[0047] Taking the steering system in a vehicle as an example, test task information is typically stored in the TMS (Test Management System). This information reflects the basic attributes, planned schedule, and actual execution status of the test task. It usually includes the test number (e.g., "TR-2024-00123"), test type (e.g., durability test, performance test, environmental test), test equipment number (e.g., "B-01"), product code under test (e.g., "1105123-01"), planned start time, planned end time, actual start time, actual end time, and test engineer identification. For tests conducted in stages (e.g., durability tests), a test stage sequence number can also be included to distinguish different test stages. For example, a durability test may include a pre-loading stage, a fatigue loading stage, and a failure diagnosis stage. Correspondingly, the test stage sequence number can be represented as "FAT-001", "FAT-002", and "FD-001".

[0048] Raw experimental data and experimental report files are typically stored in the TDMS system. These files reflect the original data collection results and conclusive outputs of the experimental process and usually include file name, file format (e.g., PDF, CSV, HDF5, or XLSX), file size, upload time, and uploader identification.

[0049] The calibration status of the test equipment used in the testing process is typically stored in the EMS system to reflect the metrological compliance and operational maintenance status of the test equipment, thereby determining whether the test equipment is in a reliable state during the test. This typically includes the latest calibration certificate number, calibration records, and maintenance records for the test equipment (e.g., test bench). Calibration records may include: test equipment number, certificate number, calibration date, and expiration date; maintenance records may include: test equipment number, maintenance date, maintenance type (e.g., periodic maintenance, fault repair, or accuracy calibration), and maintenance description.

[0050] Engineering changes to the product under test are typically stored in a PLM system to reflect the traceability status of the product under test at the design level. This usually includes the engineering change level and version number of the product under test at the current test time, specifically including the corresponding design drawing version, material specification version, and manufacturing batch number.

[0051] Defect records during the testing process are typically stored in the ITS system to reflect quality issues discovered during the testing process and their processing progress. These records usually include the defect number, severity level (e.g., fatal, severe, minor, or slight), submission time, current processing status, handler, and defect record description.

[0052] Step 202: Based on the preset test identifier field, perform association matching on the test data in each test business system to obtain the multi-source test data corresponding to each test identifier.

[0053] Here, the preset test identifier field can be used to distinguish different test tasks. In this embodiment, the test identifier field can be used as the association primary key to associate test data from different test business systems with the corresponding test identifier, eliminating the need for manual verification and matching, thereby avoiding data omissions and matching errors caused by manual operation.

[0054] For example, in this embodiment, the test number can be used as a test identifier field to perform data association matching and determine the multi-source test data corresponding to each test number.

[0055] Step 203: For the multi-source experimental data corresponding to each experimental identifier, perform multi-dimensional independent judgment according to multiple preset evaluation dimensions to obtain the initial validity results corresponding to each evaluation dimension.

[0056] Here, the preset evaluation dimensions may include: test data dimension, test equipment dimension, tested product dimension, test index dimension, and test defect dimension.

[0057] The test data dimension is used to determine the completeness of the test data; the test equipment dimension is used to determine whether the test equipment was in a valid metrological calibration state during the test; the tested product dimension is used to determine whether the design version and batch traceability information of the tested product are complete and verifiable; the test index dimension is used to determine whether the key performance indicators of the test meet the preset acceptance criteria; and the test defect dimension is used to determine whether there were any defects during the test and the status of any defect handling.

[0058] This embodiment can independently determine the validity of test results from five core levels: data, equipment, product, indicators, and defects. It comprehensively covers all key aspects of evaluating the effectiveness of test results, avoiding the bias and limitations that may result from judging from a single dimension. Furthermore, each dimension is independent and does not interfere with the others, allowing for a comprehensive multi-angle evaluation while also precisely pinpointing specific defects when problems occur in a particular dimension, facilitating targeted corrective measures in the future.

[0059] In some embodiments, the initial validity results may include: the judgment results and base scores corresponding to each evaluation dimension. The judgment results may include: pass or fail.

[0060] For each evaluation dimension, if the multi-source experimental data meets the preset conditions corresponding to that evaluation dimension, then the evaluation dimension is deemed to have passed, and the basic score corresponding to that evaluation dimension is determined; if the multi-source experimental data does not meet the preset conditions corresponding to that evaluation dimension, then the evaluation dimension is deemed to have failed, and the basic score corresponding to that evaluation dimension is determined.

[0061] The following sections will provide a detailed explanation of the pre-defined conditions and basic score determination methods for each of the five evaluation dimensions.

[0062] Here, the preset conditions for the experimental data dimensions are: multi-source experimental data includes experimental report files and raw experimental data files. The experimental report files and raw experimental data files are stored in the TDMS system.

[0063] For each test identifier, if the multi-source test data corresponding to that test identifier contains at least one test report file and at least one original test data file, then the test data dimension is considered passed. If the multi-source test data is missing a test report file or an original test data file, then the test data dimension is considered failed.

[0064] After confirming that the experimental data dimensions pass, the data sampling frequency in the original experimental data file can be further analyzed to determine the experimental data quality. If the data sampling frequency is greater than or equal to the preset sampling requirement frequency, the base score is determined as the first score (e.g., 100 points). If the data sampling frequency is less than the preset sampling requirement frequency, the base score is determined as the second score (e.g., 50 points). If the experimental data dimension fails, the base score is determined as the third score (e.g., 0 points). Here, the first score is greater than the second score, and the second score is greater than the third score.

[0065] The preset condition for the test equipment dimension is: the actual execution time of the test in the multi-source test data falls within the validity period of the test equipment's calibration certificate. The actual execution time refers to the time interval between the actual start time and the actual end time of the test. The actual start time and the actual end time are stored in the TMS system. The validity period of the test equipment's calibration certificate is the time interval between the calibration date and the expiration date; both the calibration date and the expiration date are stored in the EMS.

[0066] For each test identifier, if the actual execution time of the test in the multi-source test data corresponding to that test identifier is within the valid time interval of the calibration certificate, then the test equipment dimension is determined to pass, and the base score is determined to be the first score (e.g., 100 points). If the actual execution date of the test is not within the valid time interval of the calibration certificate, then the test equipment dimension is determined to fail, and the base score is determined to be the third score (e.g., 0 points).

[0067] The preset condition for the tested product dimension is that the multi-source test data contains all traceability information of the tested product. Here, all traceability information may include the design drawing version, material specification version, and manufacturing batch number.

[0068] For each test identifier, if the multi-source data corresponding to that test identifier contains the design drawing version, material specification version, and manufacturing batch number of the tested product, then the tested product dimension is determined to pass, and the base score is determined to be the first score. If any of the above traceability information is missing from the multi-source data, then the tested product dimension is determined to fail, and the base score is determined to be the third score.

[0069] The preset conditions for the test index dimensions are: the measured values ​​of performance indicators in the multi-source test data meet the preset acceptance criteria. The measured values ​​of performance indicators can be obtained by parsing from the original test data files. The preset acceptance criteria can be obtained from the PLM system.

[0070] For each test indicator, the measured values ​​of each performance indicator can be compared with the corresponding acceptance criteria one by one. If the measured values ​​of each performance indicator meet the corresponding acceptance criteria, the test indicator dimension is deemed to have passed, and the base score is determined to be the first score (e.g., 100 points). If the measured value of any performance indicator does not meet the preset acceptance criteria, the test indicator dimension is deemed to have failed, and the base score is determined to be the third score (e.g., 0 points).

[0071] For example, differentiated acceptance criteria can be adopted based on the importance level of the indicators. For instance, the tolerance grading information for each indicator can be read from the PLM system: the allowable deviation for critical characteristic indicators is ±2% of the benchmark value; the allowable deviation for important characteristic indicators is ±5% of the benchmark value; and the allowable deviation for general characteristic indicators is ±10% of the benchmark value.

[0072] The preset conditions for the test defect dimension are: no defect record exists in the multi-source test data, or a defect record exists in the multi-source test data and the defect record indicates that the defect processing is complete. Defect records during the test process are stored in the ITS system. For each test identifier, if no defect record exists in the multi-source test data corresponding to that test identifier, the test defect dimension is considered passed, and the base score is determined to be the first score (e.g., 100 points). If a defect record exists, the current processing status of each defect record is checked. If the current processing status indicates that the defect has been processed (e.g., the current processing status is "closed" or "verified closed"), the test defect dimension is considered passed, and the base score is determined to be the first score. If the current processing status indicates that the defect has not been processed (e.g., the current processing status is "open" or "processing"), the test defect dimension is considered failed, and the base score is determined to be the third score (e.g., 0 points).

[0073] Step 204: Using the preset cross-dimensional consistency check, the initial validity results corresponding to each evaluation dimension are corrected to obtain the test evaluation results corresponding to each test identifier.

[0074] Based on the initial validity results of each evaluation dimension, this embodiment can further utilize the logical constraint relationships between each evaluation dimension to perform cross-dimensional consistency verification, thereby correcting the initial validity results and obtaining the test evaluation results corresponding to each test identifier.

[0075] In some embodiments, the baseline scores corresponding to each evaluation dimension can be corrected using the judgment results of each evaluation dimension to determine the comprehensive score corresponding to each test identifier, and the test evaluation result can be determined based on the comprehensive score. Here, the test evaluation result can include: the test result is valid or the test result is invalid. When the comprehensive score is higher than or equal to the valid score threshold, the test result can be determined to be valid. When the comprehensive score is lower than the valid score threshold, the test result can be determined to be invalid.

[0076] This embodiment uses test identifiers to correlate and match test data scattered across various test business systems, breaking down data barriers between these systems and enabling rapid aggregation and accurate correlation of dispersed data. This overcomes the problems of low efficiency, easy omissions, and data matching errors associated with manual cross-system queries. Furthermore, by independently judging the multi-source test data corresponding to each test identifier according to multiple preset evaluation dimensions, test quality can be assessed from different perspectives, comprehensively covering various quality issues that may arise during the test and avoiding the bias that can result from single-dimensional judgments. Moreover, by using preset cross-dimensional consistency checks to correct the initial validity results of each evaluation dimension, logical conflicts and anomalies that cannot be identified when each dimension is judged individually can be discovered. The initial validity results are then adjusted in a coordinated manner to eliminate logical contradictions between dimensions, thereby significantly improving the accuracy and consistency of the test evaluation results, making the final test evaluation results more comprehensive and reliable.

[0077] See Figure 3 Next, we will elaborate on the process of correlation and matching of experimental data.

[0078] Step 301: For every two test business systems, determine whether they both contain a test identifier field.

[0079] Step 302: If at least one of the two test business systems does not contain a test identifier field, then for the test business system that does not contain a test identifier field, obtain the test data creation time and test data description in that test business system, determine the test identifier based on the test data creation time and test data description, and determine whether the test identifiers in the two test business systems correspond. If they correspond, then associate the test data in the two test business systems according to the corresponding test identifiers in the two test business systems.

[0080] Step 303: Based on the test data associated with each test business system, obtain the multi-source test data corresponding to each test identifier.

[0081] Different testing systems contain testing data records with varying dimensions. When matching testing data from different testing systems, it's possible to check whether the testing data records in each system contain a test identifier field. For testing systems that do not contain a test identifier field, the test identifier can be determined based on the test data creation time and test data description, and data association can be performed according to the test identifier.

[0082] Here, the test identifier field can be the test number. For example, defect records in the ITS system usually do not contain test numbers. However, the test number corresponding to different defect records can be determined based on the test data creation time (i.e., defect record creation time) and test data description (i.e., defect record description) in the defect records in the ITS system. Then, based on the test number, the defect records can be associated and matched with test data in other systems.

[0083] In some embodiments, the process of determining the test identifier based on the test data creation time and test data description is as follows: Based on the actual execution time of each test identifier in the test management system, the time difference between the test data creation time and the actual execution time of each test identifier is calculated. If the time difference is less than the set time difference, the test data of the test business system that does not contain the test identifier field is determined as the candidate matching item of the corresponding test identifier. Then, for each candidate matching item corresponding to each test identifier, the test data description in the candidate matching item is scanned by regular expression. If the regular expression scan result matches the test identifier, the test identifier is determined as the test identifier of the candidate matching item.

[0084] The TMS system stores task information for different experimental tasks, including experimental identifiers and the actual execution time of each experimental identifier. This embodiment uses the actual execution time of each experimental identifier in the TMS system as a benchmark to perform time window filtering to determine candidate matches.

[0085] This embodiment can calculate the time difference between the creation time of the test data and the actual execution time of the test. If the time difference is less than the set time difference (e.g., 7 working days), the corresponding test data will be identified as a candidate match for the corresponding test identifier.

[0086] Here, the actual execution time of the experiment refers to the time interval between the actual start time and the actual end time of the experiment. Considering that the experimental data is usually created after the experiment ends, this embodiment can use the time difference between the experimental data creation time and the end time of the actual execution time (i.e., the experiment end time) as the time difference between the experimental data creation time and the actual execution time, thereby accurately measuring the degree of delay of the experimental data creation time relative to the actual completion time of the experiment, and thus determining whether the experimental data is within a reasonable time window.

[0087] Taking defect records as an example, if a defect record was created on March 20, 2024, and the actual end date of a test corresponding to a certain test identifier in the test management system was March 15, 2024, the time difference between the two is 5 days. If the time difference is set to 7 working days, then the defect record can be identified as a candidate match for that test identifier. If another defect record was created on April 10, 2024, and the time difference between it and the aforementioned actual end date of the test is 26 days, then this defect record will not be included in the candidate matches.

[0088] Based on the identified candidate matches, for each test identifier, a regular expression scan is performed on the test data description of each candidate match to extract the identifier feature strings that conform to the naming convention in the test data description text, so as to determine the test identifier information hidden in the test data description.

[0089] Regular expression scanning searches the experimental data description for strings that match the experimental identifier naming format according to preset character matching rules. If the extracted identifier feature string matches a specific experimental identifier in the experimental management system, the match is considered successful, and that experimental identifier is identified as the associated experimental identifier for the candidate match. If the extracted identifier feature string does not match any of the experimental identifiers, or if a valid identifier feature string cannot be extracted from the experimental data description, the match is considered unsuccessful, and the candidate match cannot be associated with any textual semantic clues.

[0090] For example, during a durability test of a vehicle steering system, after discovering an anomaly, the test engineer typically creates a defect record in the ITS (e.g., "TR-2024-00123: Microcracks found in the housing during fatigue loading") several days after the test ends. First, a time window is selected. If the creation time of this defect record differs from the actual end time of the test with test number "TR-2024-00123" in the TMS by two working days, it is identified as a candidate match. Then, a regular expression scan is used to match strings in the test number format. If the defect record description contains "TR-2024-00123", a successful match is determined, and the test number for this defect record is assigned as TR-2024-00123.

[0091] In some embodiments, after step 301 described above, the following may also be included: Step 304: If both test business systems contain a test identifier field, determine whether the encoding format of the test identifier field in the two test business systems is the same.

[0092] Step 305: If the encoding format of the test identifier field in the two test business systems is different, then determine whether the test identifiers in the two test business systems correspond based on the string similarity between the test identifiers in the two test business systems. If they correspond, then associate the test data in the two test business systems according to the corresponding test identifiers in the two test business systems.

[0093] The test identifier field in different test operation systems may use different encoding formats. Taking the test number as an example, the test number format in the TMS system is "TR-2024-00123", while the corresponding test number format in the PLM system is "T24-00123". In such cases, cross-system data association cannot be achieved directly through equivalence matching of the test identifier field.

[0094] To address the above situation, this embodiment can calculate the string similarity between test identifiers in two test business systems, and determine whether the test identifiers in different systems point to the same test task based on the string similarity, so as to achieve fuzzy matching.

[0095] Methods for calculating string similarity may include edit distance algorithms, Damerau-Levenshtein distance algorithms, Jaro-Winkler distance algorithms, longest common substring algorithms, etc., and this embodiment does not specifically limit them. For example, this embodiment may use an edit distance algorithm to calculate the edit distance between test identifiers to reflect their string similarity.

[0096] Here, edit distance refers to the minimum number of single-character edit operations required to transform one string into another. These single-character edit operations include insertion, deletion, and replacement. A smaller edit distance indicates that the two strings are more similar.

[0097] Based on the determined edit distance, this embodiment further calculates the normalized deviation of the edit distance: in, This represents the normalization bias, used to measure the degree of difference between two test indicators, and its value ranges from 0 to 1. Indicates test identification and test marking Edit distance between Indicates test identification Length, Indicates test identification The length.

[0098] If the normalized deviation is less than the set deviation value, the fuzzy matching is successful, and the two test identifiers correspond to each other. If the normalized deviation is greater than the set deviation value, the fuzzy matching fails, and the two test identifiers do not correspond.

[0099] Here, the specific value of the deviation can be determined according to the actual situation, and this embodiment does not impose a specific limitation on it. For example, the deviation value can be set to 0.15.

[0100] If a match is successful, this embodiment can further calculate the confidence level based on the normalized bias described above. Here, the confidence level can be expressed as: 1- .

[0101] Step 306: If the encoding format of the test identifier field in the two test business systems is the same, then the test data with the same test identifier in the two test business systems are associated.

[0102] For test operation systems with the same encoding format, equi-matching can be performed directly. Here, the test identifier field can be used as the association primary key to perform equi-joins on test data records with the same test identifier in the two test operation systems, thereby associating data records belonging to the same test task under the same test identifier.

[0103] Step 307: Based on the test data associated with each test business system, obtain the multi-source test data corresponding to each test identifier.

[0104] This embodiment uses fuzzy matching or equivalent matching to associate test data corresponding to the same test identifier across different test service systems. For each test identifier, all test data associated with it in each test service system is aggregated to obtain the multi-source test data corresponding to that test identifier. The multi-source test data can cover test task information from the TMS system, raw test data and test report files from the TDMS system, test equipment calibration information from the EMS system, product under test information from the PLM system, and defect record information from the ITS system.

[0105] Here, through the aforementioned association and matching, the experimental data originally stored separately in various experimental business systems can be associated and matched according to the experimental identifier, resulting in multi-source experimental data corresponding to each experimental identifier. Based on this data association, the validity of the experimental results needs to be quantitatively evaluated. For this purpose, see [link to relevant documentation]. Figure 4 Another embodiment of this application provides a method for evaluating test results, including: Step 401: Obtain test data from multiple test business systems.

[0106] Step 402: Based on the preset test identifier field, perform association matching on the test data in each test business system to obtain the multi-source test data corresponding to each test identifier.

[0107] Step 403: For the multi-source experimental data corresponding to each experimental identifier, perform multi-dimensional independent judgment according to multiple preset evaluation dimensions to obtain the initial validity results corresponding to each evaluation dimension.

[0108] For specific implementation methods of steps 401-403 above, please refer to the above. Figures 2 to 3 The specific implementation examples are not described here.

[0109] Step 404: Based on the logical constraints between the various evaluation dimensions and the judgment results corresponding to each evaluation dimension, the basic scores corresponding to each evaluation dimension are corrected to obtain the target scores corresponding to each evaluation dimension.

[0110] Here, the judgment result can include: pass or fail. It is understood that there are logical constraints between different evaluation dimensions. Independent judgment of a single dimension only verifies the compliance of that dimension itself and cannot identify logical contradictions and credibility transmission effects between dimensions. This embodiment can modify the base score corresponding to each evaluation dimension based on the above logical constraints and the pass / fail status of each evaluation dimension, thereby enabling the target score to more accurately reflect the actual validity of the experimental data.

[0111] In some embodiments, if the test equipment dimension fails the test but the test data dimension passes the test, the base score of the test data dimension is reduced to obtain the target score of the test data dimension; if the product under test dimension fails the test, the base score of the test index dimension is reduced to obtain the target score of the test index dimension.

[0112] Here, the experimental equipment dimension is a prerequisite for the experimental data dimension. Only when the experimental equipment is operating within its calibration validity period can the collected experimental data have a reliable basis. If the experimental equipment dimension fails the assessment, even if complete experimental data exists, it cannot be used as valid evidence. Therefore, this embodiment stipulates that if the experimental equipment dimension fails the assessment, even if the experimental data dimension passes, the base score of the experimental data dimension must be reduced to obtain the target score, reflecting the decrease in reliability of this dimension due to abnormal equipment status. If the experimental equipment dimension passes, the base score of the experimental data dimension is determined as the target score.

[0113] Here, the target score for the experimental data dimension can be determined by multiplying the preset first reduction ratio by the base score of the experimental data dimension. The first reduction ratio can be determined according to the actual situation and is not specifically limited here. For example, the first reduction ratio can be 50%, that is, 50% of the base score of the experimental data dimension is determined as the target score of the experimental data dimension.

[0114] Furthermore, there is a causal constraint between the tested product dimension and the test indicator dimension. If the tested product dimension fails, it indicates that the traceability information of the tested product is incomplete, and the consistency between the product version used in the test and the design baseline cannot be confirmed. In this case, the credibility of the conclusion regarding whether the measured values ​​of the test data in the test indicator dimension meet the acceptance criteria decreases.

[0115] Therefore, in this embodiment, when the tested product dimension fails, the base score of the test indicator dimension is reduced to obtain the target score of the test indicator dimension, reflecting the decrease in the reliability of the test indicator judgment due to incomplete product traceability information. If the tested product dimension passes, the base score of the test indicator dimension can be determined as its target score.

[0116] Similarly, the target score for the experimental indicator dimension can be determined by multiplying the preset second reduction ratio by the base score of the experimental indicator dimension. The second reduction ratio can be determined according to the actual situation and is not specifically limited here. For example, the second reduction ratio can be 70%, that is, 70% of the base score of the experimental indicator dimension is determined as the target score of the experimental indicator dimension.

[0117] Furthermore, this embodiment can also perform cross-validation of defects at the test equipment level. Specifically, it can detect whether there are keywords related to the accuracy of the test equipment in the defect records. If there are defect records containing such keywords, it further detects whether the creation time of the defect record overlaps with the actual test execution time (i.e., the defect creation time is between the actual start time and the actual end time of the test). Here, the aforementioned keywords may include equipment accuracy abnormalities, sensor drift, calibration deviation, etc.

[0118] If a defect related to the accuracy of the test equipment exists, and the creation time of this defect overlaps with the actual execution time of the test, then the test equipment is determined to have an accuracy anomaly during the test. In this case, even if the test equipment is still within the calibration validity period, the test equipment dimension will be forcibly adjusted to fail, its base score will be adjusted accordingly, and the target score for the test equipment dimension will be further determined.

[0119] For each evaluation dimension, if the score of the evaluation dimension has been adjusted based on the above logical constraints, then its base score is adjusted to the target score; if the score of the evaluation dimension has not been adjusted based on the above logical constraints, then its base score is determined as its target score.

[0120] Step 405: Obtain the test type corresponding to each test identifier, and determine the weight of each evaluation dimension in each test identifier according to the test type.

[0121] Understandably, different test types place varying degrees of emphasis on each evaluation dimension. For example, durability testing has a longer cycle, and equipment performance may drift during the test; therefore, the weight of the test equipment dimension is relatively high. The core purpose of performance testing is to verify whether the product meets design specifications; therefore, the weight of the test performance dimension is relatively high. Environmental testing is greatly affected by environmental factors such as temperature and humidity, so the weight of the test equipment dimension is also relatively high. Therefore, this embodiment assigns different weight coefficients to each evaluation dimension for different test types.

[0122] For each test identifier, the corresponding test type can be obtained. Then, based on the test type, a pre-defined weight configuration table is consulted to determine the weights for each evaluation dimension. The weight configuration table pre-stores the weights for each evaluation dimension under different test types. Under the same test type, the sum of the weights for each evaluation dimension is 1.

[0123] This embodiment uses durability testing, performance testing, and environmental testing as examples to illustrate the weights of each evaluation dimension under different test types, as shown in Table 1 below: Table 1 Step 406: Based on the weights and the target scores corresponding to each evaluation dimension in each test label, determine the comprehensive score for each test label.

[0124] Here, based on the weights corresponding to each evaluation dimension, the target scores for each evaluation dimension are weighted and summed to determine the comprehensive score for each experimental label. This comprehensive score reflects the validity of the experimental results corresponding to each experimental label. It is understandable that a higher comprehensive score indicates a higher degree of validity of the corresponding experimental results.

[0125] Step 407: Determine the test evaluation result corresponding to each test label based on the comprehensive score of each test label.

[0126] This embodiment can directly determine the test evaluation result corresponding to each test identifier based on the comprehensive score. For example, when the comprehensive score is higher than or equal to the valid score threshold, the test result is determined to be valid. When the comprehensive score is lower than the valid score threshold, the test result can be determined to be invalid.

[0127] In some embodiments, in addition to determining the overall score of each test identifier, additional bonus and deduction rules are set to further optimize the overall score so that the evaluation results more comprehensively reflect the actual state of test quality.

[0128] The bonus point rules are used to positively incentivize experiments that demonstrate excellent performance in areas such as data standardization, traceability, and operational consistency. Bonus point rules may include, but are not limited to, one or more of the following: 1. The time difference between the actual start time of the test and the date of the most recent maintenance of the test equipment is less than a set time threshold (e.g., 90 days). If this rule is met, a first bonus score (e.g., 5 points) can be added to the overall score. This rule is used to reward situations where the equipment is in good maintenance condition before the test, because after recent maintenance, the equipment's operating condition and accuracy are usually at a better level, and the test data collected is more reliable.

[0129] 2. The manufacturing batch number of the tested product exists and is not empty. If this rule is met, a second bonus score (e.g., 5 points) can be added to the overall score. The existence and non-empty manufacturing batch number means that the tested product can be traced back to a specific production batch, which allows for precise containment in the event of a quality problem, thus giving a positive bonus.

[0130] 3. The uploader's identifier in the TDMS system matches the test engineer's identifier in the TMS system. If this rule is met, a third bonus score (e.g., 3 points) can be added to the overall score. This rule is used to reward situations where the test data is directly uploaded by the engineer responsible for the test, reducing the risk of version confusion or data errors that may be introduced in intermediate steps.

[0131] For each test identifier, the additional scores corresponding to all the scoring rules satisfied by that test identifier can be summed to obtain the additional score corresponding to that test identifier.

[0132] The deduction rules are used to negatively penalize experiments that have deficiencies in data quality, calibration status, process compliance, etc. The deduction rules in this embodiment may include, but are not limited to, one or more of the following: 1. If the test equipment dimension fails the assessment but the test data dimension passes, the first penalty score (e.g., 5 points) will be deducted. As mentioned earlier, the test equipment dimension is a prerequisite for the test data dimension. Expired equipment calibration will lead to a decrease in data reliability, and a deduction should be made even if the data file is complete.

[0133] 2. If the tested product dimension fails the test, a second penalty score (e.g., 3 points) will be deducted. Incomplete traceability information of the tested product will lead to a decrease in the credibility of the test indicator dimension, therefore, a deduction will be made for tests where the product dimension is judged to have failed.

[0134] 3. If a defect record related to the accuracy of the test equipment exists, and the creation time of the defect record overlaps with the actual test execution time, a third penalty score (e.g., 3 points) will be deducted. This rule is used to penalize situations where the accuracy of the test equipment fails.

[0135] For each test identifier, the penalty scores corresponding to all the deduction rules satisfied by that test identifier can be summed to obtain the penalty score corresponding to that test identifier.

[0136] This embodiment can determine the additional score and penalty score corresponding to each test identifier based on the above-mentioned scoring rules, and calculate the final score corresponding to each test identifier based on the additional score, penalty score and comprehensive score, and then determine the test evaluation result based on the final score.

[0137] Here, the final score can be expressed as: in, This indicates the final score corresponding to the test identifier. Indicates the first The weights corresponding to each evaluation dimension Indicates the first The target score corresponding to each evaluation dimension This indicates the additional points determined based on the scoring rules. This indicates the penalty score determined based on the deduction rules.

[0138] When the product under test undergoes non-phased tests such as performance tests or environmental tests, the test results can be evaluated using the above method. When the product under test undergoes phased tests such as durability tests, since there is a transfer relationship between each test phase during the phased test process (for example, the equipment calibration status, defect status, and measured performance index values ​​of the previous test will be transferred to the subsequent test), this embodiment further constructs a test chain based on the above evaluation method, and corrects the above initial validity results based on the test chain, and applies penalty points.

[0139] In some embodiments, if the test type is a durability test, the test identifiers of the durability test can be aggregated to obtain a test chain. The test chain is constructed as follows: The test identifiers are grouped according to the product code under test in the multi-source test data corresponding to each test identifier. Here, each group contains multiple test identifiers, and multiple test identifiers correspond to the same product code under test. For example, if multiple test identifiers correspond to the same product code under test, these test identifiers are grouped into the same group.

[0140] Next, for each group, the test plans are sorted in ascending order according to the start time of each test identifier to obtain the test sequence corresponding to that group, thus establishing the temporal order of each test task. For example, if a group contains four test identifiers, the order after sorting by planned start time is: TR-2024-00123 (planned start time 2024-03-01), TR-2024-00124 (planned start time 2024-03-02), TR-2024-00125 (planned start time 2024-03-05), and TR-2024-00126 (planned start time 2024-03-06).

[0141] Subsequently, the test sequence is traversed. If the test time interval between two adjacent test identifiers is less than a set time interval, and the test equipment corresponding to the two adjacent test identifiers is the same, then the two adjacent test identifiers are determined to satisfy the aggregation relationship. Here, the set time interval can be determined based on the rest window duration of each adjacent test stage during the durability test. For example, it can be 72 hours.

[0142] Using the example above: TR-2024-00123 and TR-2024-00124 have a time interval of 1 day ≤ 72 hours and use the same test bench (both B-01), satisfying the aggregation relationship; TR-2024-00124 and TR-2024-00125 have a time interval of 3 days ≤ 72 hours, but if the test benches are different (the former is B-01, the latter B-02), then the aggregation relationship is not satisfied. Here, the test equipment used in the durability test is the test bench.

[0143] Finally, multiple test identifiers that consecutively satisfy the aggregation relationship in the test sequence are aggregated into the same test chain.

[0144] Traverse the entire test sequence, starting from the first test identifier and checking the aggregation relationship sequentially. If two adjacent test identifiers satisfy the aggregation relationship, they are included in the same test chain; if an adjacent pair that does not satisfy the aggregation relationship is found, the test chain is broken at that point, and the current test identifier is used as the starting point of a new chain to continue traversing, resulting in all test chains.

[0145] After the above aggregation is completed, each test chain contains multiple test identifiers arranged in chronological order and their corresponding multi-source test data. It is understood that there are state transitive relationships between the test identifiers within the same test chain. To further improve the accuracy of the evaluation results, this embodiment can correct the initial validity judgment results of each test identifier within the same test chain based on the state transitive relationships within the test chain.

[0146] Specifically, after performing multi-dimensional independent judgments on each test identifier according to step 403 above, this embodiment can also correct the initial validity results of the test identifiers based on the chain transmission order of each test identifier in the test chain. The correction rules are as follows: For each test chain, the validity period of the calibration certificate for the first test identifier in that chain can be passed on to subsequent test identifiers. If the actual execution time of subsequent test identifiers exceeds the validity period of the calibration certificate, the test equipment dimension of the subsequent test identifiers is determined to be unsuccessful, and the base score for the test equipment dimension is recalculated. This rule ensures that all tests in the same test chain are completed within the same equipment calibration validity period, avoiding inconsistencies in equipment status caused by test chains spanning calibration cycles.

[0147] For each test identifier in each test chain, if the test defect dimension judgment of any preceding test identifier fails, then the test defect dimension judgment of that test identifier also fails, thus determining the judgment result and base score for the test defect dimension. Here, preceding test identifiers refer to all test identifiers in the test chain that precede this test identifier. This rule is used to ensure that quality issues left over from previous tests are not ignored as the test phase progresses, preventing continued testing with defects.

[0148] For each test marker in each test chain, the measured values ​​of each performance indicator in each preceding test marker are determined as the benchmark values ​​in the acceptance criteria for that test marker. These benchmark values ​​are used to check whether the test indicator dimension of that test marker passes, thereby determining the judgment result and basic score for that test indicator dimension. This rule can detect equipment performance drift by comparing the changing trends of measured values ​​of preceding and subsequent tests within the same chain. Here, for each performance indicator, the average of the measured values ​​of that performance indicator in each preceding test can be used as the benchmark value in the acceptance criteria. If subsequent tests show a significant deviation from preceding tests, it indicates that the equipment may have accuracy drift or the tested component may have performance degradation.

[0149] Based on step 403, this embodiment further modifies the initial validity results obtained by each test identifier in step 403 based on the state transmission status between each test identifier in the test chain, resulting in modified initial validity results. Next, based on the modified initial validity results, the cross-dimensional consistency check described in step 404 is performed to determine the target score for each evaluation dimension, and further determine the comprehensive score for each test identifier.

[0150] This embodiment can also, based on the overall score, penalize and deduct points according to the order of each test identifier in the test chain and the overall score corresponding to each test identifier, thereby passing the failure status of the previous test to the subsequent test and preventing the test from continuing with defects.

[0151] Specifically, for each test chain, based on the order of the test identifiers in the test chain and the comprehensive score corresponding to each test identifier in the test chain, the penalty deduction value corresponding to each test identifier in the test chain is determined; based on the comprehensive score and penalty deduction value of each test identifier, the test evaluation result corresponding to each test identifier is determined.

[0152] Here, the specific process for determining the penalty points is as follows: For each test marker in each test chain, check whether the overall score of the preceding test marker is lower than a set score. If the overall score of the preceding test marker is lower than the set score, calculate the difference between the overall score of the preceding test marker and the set score. Based on the difference, determine the penalty score corresponding to the test marker. The higher the difference, the higher the penalty score.

[0153] If the overall score is lower than the set score, the larger the difference between the overall score and the set score, the higher the risk of the test result being invalid. Within the same test chain, if a preceding test fails, subsequent tests need to be penalized by deducting points to transfer the risk of failure from the preceding test to subsequent tests and prevent testing from continuing with a faulty result. Here, the penalty point value reflects the degree to which the validity of subsequent test results is reduced due to the failure of the preceding test.

[0154] For each test marker in the test chain, a tiered deduction method can be used to determine the penalty points. Specifically, for each preceding test marker, based on the difference between the overall score and the set score of the preceding test marker, the deduction point corresponding to the preceding test is determined according to the tier of the difference. Here, different tiers correspond to different deduction points. For example, 2 points are deducted for every 20-point difference, and any difference less than 20 points can be discarded. For instance, when the difference is 50 points, 4 points are deducted.

[0155] After determining the deduction value corresponding to each preceding test identifier, for the current test identifier, the deduction values ​​corresponding to each preceding test identifier are accumulated to obtain the target deduction value.

[0156] To avoid excessive deductions in subsequent tests due to excessively long chains or low scores in preceding tests, this embodiment sets a maximum deduction score and determines the minimum of the target deduction score and the maximum deduction score as the penalty deduction score corresponding to the test identifier.

[0157] Here, the set score and the maximum deduction score can be determined according to the actual situation, and this embodiment does not impose specific limitations on them. For example, the set score can be 80 points, and the maximum deduction score can be 5 points.

[0158] Based on the overall score and penalty deduction, this embodiment can further combine the additional score and penalty score determined based on the above-mentioned additional scoring rules and deduction rules to determine the final score corresponding to each test identifier, and determine the test evaluation result based on the final score.

[0159] Here, the final score can be expressed as: in, This indicates the final score corresponding to the test identifier. Indicates the first The weights corresponding to each evaluation dimension Indicates the first The target score corresponding to each evaluation dimension This indicates the additional points determined based on the scoring rules. This indicates the penalty score determined based on the deduction rules. This represents the penalty deduction value determined based on the chain of trials.

[0160] Understandably, when the final score is higher than or equal to the valid score threshold, the test result is considered valid. When the overall score is lower than the set score, the test result is considered invalid. Here, the valid score threshold can be determined according to the actual situation; for example, the valid score threshold can be 80 points.

[0161] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0162] Figure 5 This is a schematic diagram of the structure of an experimental result evaluation device provided in an embodiment of this application. Figure 5 As shown, the test result evaluation device provided in this embodiment may include: an acquisition module 501, a matching module 502, and an evaluation module 503.

[0163] The acquisition module 501 is used to acquire test data from multiple test business systems.

[0164] The matching module 502 is used to perform association matching on the test data in each test business system based on the preset test identifier field to obtain the multi-source test data corresponding to each test identifier.

[0165] Evaluation module 503 is used for: For the multi-source experimental data corresponding to each experimental identifier, multi-dimensional independent judgment is performed according to multiple preset evaluation dimensions to obtain the initial validity results corresponding to each evaluation dimension; By using a pre-defined cross-dimensional consistency check, the initial validity results corresponding to each evaluation dimension are corrected to obtain the test evaluation results corresponding to each test identifier.

[0166] In one possible implementation, the matching module 502 is specifically used for: For every two trial business systems, determine whether they both contain a trial identifier field; If at least one of the two test business systems does not contain a test identifier field, then for the test business system that does not contain a test identifier field, the test data creation time and test data description in that test business system are obtained. Based on the test data creation time and test data description, the test identifier is determined, and it is determined whether the test identifiers in the two test business systems correspond. If they correspond, the test data in the two test business systems are associated according to the corresponding test identifiers in the two test business systems. Based on the test data associated with each test business system, multi-source test data corresponding to each test identifier is obtained.

[0167] In one possible implementation, the matching module 502 is also used for: If both test business systems contain a test identifier field, then determine whether the encoding format of the test identifier field in the two test business systems is the same; If the encoding format of the test identifier field in the two test business systems is different, the string similarity between the test identifiers in the two test business systems is used to determine whether the test identifiers in the two test business systems correspond. If they correspond, the test data in the two test business systems are associated according to the corresponding test identifiers in the two test business systems. If the encoding format of the test identifier field in two test business systems is the same, then the test data with the same test identifier in the two test business systems will be associated. Based on the test data associated with each test business system, multi-source test data corresponding to each test identifier is obtained.

[0168] In one possible implementation, the test operation system includes a test management system; the test management system includes a test identifier field; the test data in the test management system includes the actual execution time of the test corresponding to each test identifier; Matching module 502 is specifically used for: Based on the actual execution time of each test identifier in the test management system, calculate the time difference between the test data creation time and the actual execution time of each test identifier. If the time difference is less than the set time difference, the test data of the test business system that does not contain the test identifier field will be identified as the candidate match for the corresponding test identifier. For each candidate matching item corresponding to each test identifier, a regular expression scan is performed on the test data description in the candidate matching item. If the regular expression scan result matches the test identifier, the test identifier is determined as the test identifier of the candidate matching item.

[0169] In one possible implementation, the initial validity results for each evaluation dimension include: the judgment result and the base score for each evaluation dimension; Evaluation module 503 is specifically used for: Based on the logical constraints between the various evaluation dimensions and the judgment results corresponding to each evaluation dimension, the basic scores corresponding to each evaluation dimension are corrected to obtain the target scores corresponding to each evaluation dimension. Obtain the test type corresponding to each test identifier, and determine the weight of each evaluation dimension in each test identifier based on the test type; Based on the weights and the target scores corresponding to each evaluation dimension in each test label, the comprehensive score of each test label is determined. Based on the comprehensive score of each test label, the test evaluation result corresponding to each test label is determined.

[0170] In one possible implementation, the test type includes durability testing; If the test type corresponding to each test identifier is a durability test, evaluation module 503 is specifically used for: The test identifiers are aggregated to obtain a test chain; the test chain contains multiple test identifiers arranged in sequence. For each test chain, based on the order of the test identifiers in the test chain and the comprehensive score corresponding to each test identifier in the test chain, the penalty deduction value corresponding to each test identifier in the test chain is determined; Based on the comprehensive score and penalty points of each test label, the corresponding test evaluation result is determined.

[0171] In one possible implementation, the evaluation module 503 is specifically used for: According to the product code of the test in the multi-source test data corresponding to each test identifier, the test identifiers are grouped; each group contains multiple test identifiers, and multiple test identifiers correspond to the same product code. For each group, sort them in ascending order according to the start time of the test plan corresponding to each test identifier to obtain the test sequence corresponding to that group; Traverse the test sequence. If the test time interval between two adjacent test identifiers is less than the set time interval, and the test equipment corresponding to the two adjacent test identifiers is the same, then it is determined that the two adjacent test identifiers satisfy the aggregation relationship. Multiple test identifiers that consecutively satisfy the aggregation relationship in the test sequence are aggregated into the same test chain.

[0172] In one possible implementation, the evaluation module 503 is specifically used for: For each test identifier in each test chain, check whether the overall score of the preceding test identifier is lower than the set score; If the overall score of the preceding test identifier is lower than the set score, then the difference between the overall score of the preceding test identifier and the set score is calculated. Based on the difference, determine the penalty points corresponding to the test label; the higher the difference, the higher the penalty points.

[0173] In one possible implementation, the preset evaluation dimensions include: test data dimension, test equipment dimension, tested product dimension, and test index dimension; the judgment result includes: pass or fail. Evaluation module 503 is specifically used for: If the test equipment dimension fails the test but the test data dimension passes the test, the base score of the test data dimension will be reduced to obtain the target score of the test data dimension. If the tested product fails the evaluation in a certain dimension, the base score for that dimension will be reduced to obtain the target score for that dimension.

[0174] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0175] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 6 As shown, the electronic device of this embodiment includes a processor 610 and a memory 620, wherein the memory 620 stores a computer program 621 that can run on the processor 610. When the processor 610 executes the computer program 621, it implements the steps in any of the above-described method embodiments, for example... Figure 2 Steps 201 to 204 are shown. Alternatively, when processor 610 executes computer program 621, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 5 The functions of modules 501 to 503 are shown.

[0176] For example, computer program 621 may be divided into one or more modules / units, one or more of which are stored in memory 620 and executed by processor 610 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 621 in an electronic device.

[0177] Those skilled in the art will understand that Figure 6 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or combinations of certain components, or different components, such as input / output devices, network access devices, buses, etc.

[0178] The processor 610 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0179] The memory 620 can be an internal storage unit of the electronic device, such as a hard drive or memory, or an external storage device, such as a plug-in hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card. The memory 620 can also include both internal and external storage units. The memory 620 is used to store computer programs and other programs and data required by the electronic device. The memory 620 can also be used to temporarily store data that has been output or will be output.

[0180] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0181] An embodiment of this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for evaluating test results.

[0182] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0183] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0184] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0185] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0186] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0187] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0188] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for evaluating test results, characterized in that, include: Acquire test data from multiple test operation systems; Based on the preset test identifier field, the test data in each test business system are correlated and matched to obtain the multi-source test data corresponding to each test identifier; For the multi-source experimental data corresponding to each experimental identifier, multi-dimensional independent judgment is performed according to multiple preset evaluation dimensions to obtain the initial validity results corresponding to each evaluation dimension; By using a preset cross-dimensional consistency check, the initial validity results corresponding to each evaluation dimension are corrected to obtain the test evaluation results corresponding to each test identifier.

2. The method for evaluating test results according to claim 1, characterized in that, The method involves associating and matching test data from various test service systems based on a preset test identifier field to obtain multi-source test data corresponding to each test identifier, including: For every two trial business systems, determine whether they both contain a trial identifier field; If at least one of the two test business systems does not contain a test identifier field, then for the test business system that does not contain a test identifier field, the test data creation time and test data description in that test business system are obtained. Based on the test data creation time and the test data description, the test identifier is determined, and it is determined whether the test identifiers in the two test business systems correspond. If they correspond, the test data in the two test business systems are associated according to the corresponding test identifiers in the two test business systems. Based on the test data associated with each test business system, multi-source test data corresponding to each test identifier is obtained.

3. The method for evaluating test results according to claim 2, characterized in that, After determining whether all tests include the test identifier field, the process also includes: If both test business systems contain a test identifier field, then determine whether the encoding format of the test identifier field in the two test business systems is the same; If the encoding format of the test identifier field in the two test business systems is different, the string similarity between the test identifiers in the two test business systems is used to determine whether the test identifiers in the two test business systems correspond. If they correspond, the test data in the two test business systems are associated according to the corresponding test identifiers in the two test business systems. If the encoding format of the test identifier field in two test business systems is the same, then the test data with the same test identifier in the two test business systems will be associated. Based on the test data associated with each test business system, multi-source test data corresponding to each test identifier is obtained.

4. The method for evaluating test results according to claim 2, characterized in that, The testing business system includes a testing management system; the testing management system includes a testing identifier field; the testing data in the testing management system includes the actual execution time of the test corresponding to each testing identifier; The process of determining the test identifier based on the test data creation time and the test data description includes: Based on the actual execution time of each test identifier in the test management system, calculate the time difference between the test data creation time and the actual execution time of each test identifier. If the time difference is less than the set time difference, then the test data of the test business system that does not contain the test identifier field will be determined as the candidate matching item for the corresponding test identifier. For each candidate matching item corresponding to each test identifier, a regular expression scan is performed on the test data description in the candidate matching item. If the regular expression scan result matches the test identifier, then the test identifier is determined as the test identifier of the candidate matching item.

5. The method for evaluating test results according to any one of claims 1 to 4, characterized in that, The initial validity results corresponding to each evaluation dimension include: the judgment result and basic score corresponding to each evaluation dimension; The process of using a preset cross-dimensional consistency check to correct the initial validity results corresponding to each evaluation dimension, and obtaining the test evaluation results corresponding to each test identifier, includes: Based on the logical constraints between the various evaluation dimensions and the judgment results corresponding to each evaluation dimension, the basic scores corresponding to each evaluation dimension are corrected to obtain the target scores corresponding to each evaluation dimension. Obtain the test type corresponding to each test identifier, and determine the weight of each evaluation dimension in each test identifier based on the test type; Based on the weights and the target scores corresponding to each evaluation dimension in each test identifier, the comprehensive score of each test identifier is determined. Based on the comprehensive score of each test label, the test evaluation result corresponding to each test label is determined.

6. The method for evaluating test results according to claim 5, characterized in that, The types of tests include durability tests; If the test type corresponding to each test label is a durability test, the determination of the test evaluation result corresponding to each test label based on the comprehensive score of each test label includes: The test identifiers are aggregated to obtain a test chain; the test chain contains multiple test identifiers arranged in sequence. For each test chain, based on the order of the test identifiers in the test chain and the comprehensive score corresponding to each test identifier in the test chain, the penalty deduction value corresponding to each test identifier in the test chain is determined; Based on the comprehensive score of each test identifier and the penalty deduction value, the corresponding test evaluation result for each test identifier is determined.

7. The method for evaluating test results according to claim 6, characterized in that, The aggregation of each test identifier to obtain the test chain includes: According to the product code under test in the multi-source test data corresponding to each test identifier, the test identifiers are grouped; each group contains multiple test identifiers, and the multiple test identifiers correspond to the same product code under test. For each group, sort them in ascending order according to the start time of the test plan corresponding to each test identifier to obtain the test sequence corresponding to that group; Traverse the test sequence. If the test time interval between two adjacent test identifiers is less than the set time interval, and the test equipment corresponding to the two adjacent test identifiers is the same, then it is determined that the two adjacent test identifiers satisfy the aggregation relationship. Multiple test identifiers that consecutively satisfy the aggregation relationship in the test sequence are aggregated into the same test chain.

8. The method for evaluating test results according to claim 6, characterized in that, For each test chain, based on the order of the test identifiers in the chain and the comprehensive score corresponding to each test identifier, the penalty deduction value corresponding to each test identifier in the chain is determined, including: For each test identifier in each test chain, check whether the overall score of the preceding test identifier is lower than the set score; If the overall score of the preceding test identifier is lower than the set score, then the difference between the overall score of the preceding test identifier and the set score is calculated. Based on the difference, the penalty score corresponding to the test identifier is determined; the higher the difference, the higher the penalty score.

9. The method for evaluating test results according to claim 5, characterized in that, The preset evaluation dimensions include: test data dimension, test equipment dimension, tested product dimension, and test index dimension; the judgment result includes: pass or fail. The process involves correcting the base scores for each evaluation dimension based on the logical constraints between the various evaluation dimensions and the judgment results corresponding to each evaluation dimension, thereby obtaining the target scores for each evaluation dimension. This includes: If the test equipment dimension fails the test but the test data dimension passes the test, the base score of the test data dimension is reduced to obtain the target score of the test data dimension. If the tested product dimension fails the test, the base score of the test indicator dimension is reduced to obtain the target score of the test indicator dimension.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the test result evaluation method as described in any one of claims 1 to 9.

Images