Online-offline based hardware platform performance test method and device and computer equipment

Abstract

The disclosure provides an online-offline-based hardware platform performance testing method and device and computer equipment, relates to the technical field of computers, in particular to the fields of computer system testing and performance evaluation, and the implementation scheme is as follows: collecting multi-channel data of a first hardware platform running in an online environment; performing abnormal data identification processing on the multi-channel data to obtain a first sub-multi-channel data segment including abnormal data; generating a scene description script based on the data of each channel and the time stamp in the first sub-multi-channel data segment; loading the scene description script to a second hardware platform to be tested for offline playback testing; performing consistency calculation on a first running result obtained by operating the first hardware platform and a second running result obtained by operating the second hardware platform to generate a consistency evaluation score; and evaluating the performance of the second hardware platform based on the consistency evaluation score to obtain a test result, so that the performance of the hardware platform can be tested.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, particularly to the fields of computer system testing and performance evaluation, and especially to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for online-offline hardware platform performance testing. Background Technology

[0002] With the widespread application of computer systems and algorithm platforms in high-performance computing scenarios such as autonomous driving and industrial control, the performance stability of hardware platforms under complex operating environments has gradually become a key concern. Current hardware platform performance testing typically relies on offline test data construction or stress testing to verify system throughput and response performance. However, these testing methods struggle to realistically reproduce the operational characteristics of online environments, such as concurrent input of multi-source data and environmental fluctuations, leading to discrepancies between test results and actual operating conditions. Therefore, how to support the reproduction of real-world operating scenarios during performance testing and evaluate hardware platform performance based on the results has become an important technological direction. Summary of the Invention

[0003] This disclosure provides a method, apparatus, computer device, computer-readable storage medium, and computer program product for online-offline hardware platform performance testing.

[0004] According to one aspect of this disclosure, a hardware platform performance testing method based on online-offline interaction is provided, comprising: collecting multi-channel data of a first hardware platform running in an online environment, wherein the multi-channel data is associated with multiple channels, and the multi-channel data includes first-channel data for recording the processing of business requests by the first hardware platform, second-channel data acquired from sensors, and third-channel data reflecting the hardware resource consumption status; performing abnormal data identification processing on the multi-channel data to obtain a first sub-multi-channel data segment including abnormal data; and generating a scenario description script based on the data of each channel in the first sub-multi-channel data segment and a timestamp, wherein the scenario description script includes first description information, second description information, and third description information, the first description information being... The first description information describes the first moment when data for each channel in the first sub-multi-channel data segment is input; the second description information describes the amount of input data for each channel in the first sub-multi-channel data segment at the first moment; and the third description information describes external interference factors in the online environment. The scenario description script is loaded onto the second hardware platform under test for offline playback testing. The second hardware platform includes a playback engine configured to parse and run the parsed scenario description script. A consistency calculation is performed on the first running result obtained from operating the first hardware platform and the second running result obtained from operating the second hardware platform to generate a consistency evaluation score. The performance of the second hardware platform is evaluated based on the consistency evaluation score to obtain the test results.

[0005] In some embodiments, collecting multi-channel data from a first hardware platform running in an online environment includes: performing time alignment processing on the first channel data, the second channel data, and the third channel data based on their respective timestamps.

[0006] In some embodiments, loading the scene description script onto the second hardware platform to be tested for offline playback testing includes: inputting data of each channel in the first sub-multi-channel data segment into the second hardware platform based on the first description information and the second description information; and adding external interference factors to the second hardware platform based on the third description information.

[0007] In some embodiments, a consistency calculation is performed on the first operating result obtained by operating the first hardware platform and the second operating result obtained by operating the second hardware platform to generate a consistency evaluation score. This includes: calculating a distribution consistency score, a time-series consistency score, an event location consistency score, and a service output consistency score between the first and second operating results. The distribution consistency score characterizes the degree of matching between the data distribution in the offline playback test environment and the data distribution in the online environment; the time-series consistency score verifies the accuracy of the amount of data output by each channel at a specific time; the event location consistency score characterizes the temporal consistency between the data output by each channel in the offline playback test environment and the data output in the online environment; and the service output consistency score verifies the accuracy of the calculation results output by the second hardware platform in the offline playback test environment. The distribution consistency score, time-series consistency score, event location consistency score, and service output consistency score are then weighted and summed according to a preset weight ratio to generate the consistency evaluation score.

[0008] In some embodiments, the performance of the second hardware platform is evaluated based on the consistency evaluation score to obtain test results, including: determining that the performance of the second hardware platform is qualified in response to the consistency evaluation score being greater than or equal to a first preset threshold; performing supplementary testing on the second hardware platform in response to the consistency evaluation score being less than the first preset threshold but greater than or equal to a second preset threshold; and determining that the performance of the second hardware platform is unqualified in response to the consistency evaluation score being less than the second preset threshold.

[0009] In some embodiments, the online-offline based hardware platform performance testing method further includes: in response to a test result indicating supplementary testing or failure, identifying a second sub-multichannel data segment in which the second hardware platform exhibits a consistency deviation during offline playback testing; and analyzing and processing the data of each channel in the second sub-multichannel data segment to generate an evaluation report for auditing.

[0010] According to one aspect of this disclosure, an online-offline based hardware platform performance testing device is provided, comprising: a data acquisition module configured to acquire multi-channel data of a first hardware platform running in an online environment, wherein the multi-channel data is associated with multiple channels, and the multi-channel data includes first channel data for recording the processing of business requests by the first hardware platform, second channel data acquired from sensors, and third channel data reflecting the hardware resource consumption status; a data identification module configured to perform abnormal data identification processing on the multi-channel data to obtain a first sub-multi-channel data segment including abnormal data; and a script generation module configured to generate a scenario description script based on the data of each channel in the first sub-multi-channel data segment and a timestamp, wherein the scenario description script includes first description information, second description information, and third description information, the first description information being used for... The system comprises three parts: a first description of the input time of each channel in the first sub-multi-channel data segment; a second description of the input data volume of each channel in the first sub-multi-channel data segment at the first time; and a third description of external interference factors in the online environment. A script loading module is configured to load the scene description script onto the second hardware platform under test for offline playback testing. The second hardware platform includes a playback engine configured to parse and run the scene description script. A score generation module is configured to perform consistency calculations on the first running result obtained from operating the first hardware platform and the second running result obtained from operating the second hardware platform to generate a consistency evaluation score. Finally, a result determination module is configured to evaluate the performance of the second hardware platform based on the consistency evaluation score to obtain the test result.

[0011] According to another aspect of this disclosure, a computer device is provided, comprising: at least one processor; and a memory having a computer program stored thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the methods provided above in this disclosure.

[0012] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0013] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the methods provided above in this disclosure.

[0014] According to one or more embodiments of this disclosure, it is possible to reproduce online operating scenarios in an offline testing environment and to quantitatively evaluate hardware performance based on online and offline operating results.

[0015] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0016] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0017] Figure 1 This is a flowchart illustrating an online-offline based hardware platform performance testing method according to an exemplary embodiment.

[0018] Figure 2 This is a schematic diagram illustrating a scene description script according to an exemplary embodiment.

[0019] Figure 3 This is a schematic diagram illustrating a consistency score weighted sum according to an exemplary embodiment.

[0020] Figure 4 This is a schematic block diagram illustrating an online-offline based hardware platform performance testing apparatus according to an exemplary embodiment.

[0021] Figure 5 An example computer device is shown in which any of the embodiments described herein may be implemented. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0024] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0025] In related technologies, performance testing of hardware platform systems is typically conducted through offline simulation playback, stress testing, or benchmarking to verify the system's operational stability and processing capabilities under different load conditions. However, these testing methods are generally based on manually constructed test data, making it difficult to reflect the system's operational characteristics in a real online environment.

[0026] First, existing offline playback technologies generally suffer from low environmental simulation fidelity. Online hardware platforms are simultaneously affected by multiple data inputs and dynamic environmental changes during actual operation, while offline testing typically focuses only on single-dimensional input data or static configuration parameters, lacking analysis of the actual runtime sequence and the correlation between multi-channel data, making it difficult to reproduce the synergistic effect of various data in the online environment over time.

[0027] Secondly, in terms of evaluating test results, related technologies often focus on single metrics such as throughput, latency, or resource utilization, lacking consistency analysis between online and offline results. Especially in scenarios involving complex business logic or multi-channel data processing, existing evaluation methods prioritize the correctness of hardware output results, failing to assess whether offline test results truly reflect the online operating status from multiple perspectives, including data distribution, timing relationships, and business output.

[0028] To address this, embodiments of this disclosure propose a more effective hardware platform performance testing method. By constructing a scenario description script that includes multi-channel business data and external environmental interference, and performing unified playback on an offline testing platform, the running results of different hardware platforms can be compared and analyzed based on multiple dimensions such as distribution, timing, and event location while maintaining runtime consistency. This improves the accuracy and reliability of hardware performance test results.

[0029] Exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0030] Figure 1 This is a flowchart illustrating an online-offline based hardware platform performance testing method according to an exemplary embodiment.

[0031] like Figure 1As shown, the online-offline based hardware platform performance testing method includes steps S101 to S106.

[0032] In step S101, multi-channel data of the first hardware platform running in the online environment is collected. The multi-channel data is associated with multiple channels and includes first-channel data for recording the first hardware platform's processing of business requests, second-channel data obtained from sensors, and third-channel data reflecting the hardware resource consumption status.

[0033] In the example, the online environment can refer to the production environment where the first hardware platform is actually deployed and processes real business operations, such as the road driving scenario of autonomous vehicles or the data center operation scenario of high-performance servers. The first hardware platform can refer to the physical device running in the online environment, which can carry out business processing tasks in the real online environment, such as a hardware computing platform composed of chips, peripherals, etc. During operation, the first hardware platform processes various input data. In order to accurately reproduce the operating state of the first hardware platform in a real business scenario, various data involved in the operation of the first hardware platform can be collected, i.e., multi-channel data.

[0034] In the example, multi-channel data can be collected from multiple channels, referring to a time-correlated collection of data acquired in parallel from different dimensions. This data is not a single log stream but includes data from various aspects such as business logic, external perception, and internal status. Multi-channel data can include three types of channel data: the first channel records specific business requests processed by the hardware platform, such as algorithm inference task requests or database query requests; the second channel includes environmental input data acquired in real time from LiDAR, cameras, or other IoT sensors; and the third channel reflects the resource consumption status of the underlying hardware, such as CPU (Central Processing Unit) utilization, memory bandwidth usage, and GPU (Graphics Processing Unit) memory status. Therefore, through the coordinated acquisition of these three types of channel data, the operating status of the first hardware platform in a real business scenario can be simulated.

[0035] In the example, data generated during the operation of the first hardware platform can be collected while it is running online. Specifically, this can be achieved by accessing the business processing module to obtain first-channel data for recording the arrival and response of business requests; by monitoring the sensor interface to obtain second-channel data generated by sensors and input to the first hardware platform; and simultaneously, by using a resource monitoring component or system interface to collect third-channel data reflecting the usage of processor, memory, or other hardware resources.

[0036] In step S102, abnormal data identification processing is performed on the multi-channel data to obtain a first sub-multi-channel data segment including abnormal data.

[0037] In the example, since the online data collection contains a large amount of stable operating data with no sampling value, directly utilizing all of it would waste computing power. Therefore, representative abnormal data in the multi-channel data can be identified and processed to extract valuable data, namely the first sub-multi-channel data segment. In specific implementation, the multi-channel trajectory data can be scanned in time sequence to match preset key scenario triggering characteristics, such as a sudden increase in the number of business requests in the first channel, a large increase in the amount of data acquired by the sensors in real time in the second channel, a peak in hardware resource consumption in the third channel, and abnormal interruption of specific business sessions. Once the preset key scenario triggering characteristics are identified, the time of the anomaly can be locked, and the interception window can be determined by extending forward and backward according to the preset time span parameters. Then, all channel data within the window can be synchronously extracted from the original multi-channel data to generate the first sub-multi-channel data segment.

[0038] In step S103, a scenario description script is generated based on the data and timestamp of each channel in the first sub-multi-channel data segment. The scenario description script includes first description information, second description information and third description information. The first description information is used to describe the first moment when the data of each channel in the first sub-multi-channel data segment is input. The second description information is used to describe the amount of input data of each channel in the first sub-multi-channel data segment at the first moment. The third description information is used to describe external interference factors in the online environment.

[0039] In the example, the scene description script can be a standardized intermediate description file, such as JSON, which does not depend on the specific underlying hardware implementation. Instead, it is indexed by time sequence and is used to reflect the input of data from different channels at a specific moment and the corresponding operating environment characteristics. The scene description script can include three types of descriptive information. During generation, firstly, based on the data of each channel in the first sub-multi-channel data segment and its corresponding timestamp, the data of each channel is organized according to the timestamp order. The time when each channel's data was input, i.e., the first moment, is determined based on the timestamp information. This moment, along with the corresponding input data channel, is written into the scene description script as the first descriptive information. This first descriptive information ensures that data input strictly follows the chronological order of the online environment during subsequent playback. Secondly, based on the data content of each channel at the same first moment, the amount of input data for each channel at that moment can be determined, such as the size of the data packet and the effective payload, which can be used as the second descriptive information. Simultaneously, combined with the online operating status of the first sub-multi-channel data segment, data features reflecting external environmental interference factors are extracted. For example, the CPU utilization rate, peak memory usage, or network jitter characteristics at that time can be extracted from the third channel data to generate the third descriptive information, such as a 100ms network delay occurring at the 5th second.

[0040] In the example, by writing the above three types of descriptive information into the scenario description script, the operating scenario of the first hardware platform within a specific time period can be abstracted and expressed while maintaining the temporal characteristics of the data, thereby providing reliable data for subsequent playback tests.

[0041] In step S104, the scene description script is loaded onto the second hardware platform to be tested for offline playback testing. The second hardware platform includes a playback engine, which is configured to parse the scene description script and run the parsed scene description script.

[0042] In the example, loading the scenario description script and performing offline playback testing can be done by launching a playback engine pre-installed on the second hardware platform. The playback engine can be a lightweight execution agent or dedicated test driver software deployed on the device under test (DUT), and may include a time driver, data injector, and environment simulator to implement offline playback testing. When the scenario description script is loaded onto the second hardware platform, the playback engine first parses the script, mapping the first, second, and third description information into a sequence of instructions. Then, it drives the second hardware platform to execute the corresponding data input process according to this sequence of instructions, introducing external environmental factors described in the script during execution, so that the second hardware platform runs offline according to a predetermined timing sequence. The playback engine does not directly participate in business logic judgment; instead, it acts as an execution carrier, scheduling the operation of the second hardware platform based on the data input time, input data volume, and external environmental information described in the scenario description script, thereby constructing a test scenario in the offline environment that corresponds to the online operation process. In this way, the second hardware platform can reproduce the operating scenario within a specific time period without relying on the real online environment, thereby realizing offline playback testing of the performance of the second hardware platform and improving the controllability and repeatability of the testing process.

[0043] In step S105, a consistency calculation is performed on the first running result obtained by operating the first hardware platform and the second running result obtained by operating the second hardware platform to generate a consistency evaluation score.

[0044] In the example, "operation" can refer to running the same version of the algorithm on both the first and second hardware platforms. In performance testing, the results of offline replay tests do not represent the results in a real-world environment. Therefore, only by quantifying the consistency between the second running result and the first running result in terms of timing and resource characteristics through consistency calculations can we determine whether the offline replay test environment truly replicates the complex online situation. After the offline replay test, the response data generated by the first hardware platform in the real online scenario (i.e., the first running result) can be used as a benchmark and time-aligned with the response data generated by the second hardware platform in the offline replay environment (i.e., the second running result). Subsequently, the deviations between the two in terms of data distribution, key event triggering timing, and business logic output can be calculated. These multi-dimensional deviations are then normalized and weighted to generate a numerical value that intuitively reflects consistency—the consistency evaluation score—providing a basis for subsequent objective evaluation of the second hardware platform's performance.

[0045] In step S106, the performance of the second hardware platform is evaluated based on the consistency evaluation score to obtain the test results.

[0046] In the example, the simulation confidence of the offline playback environment can be verified by the consistency evaluation score. If the score is higher than the preset threshold, it indicates that the offline test largely reproduces the online environment, and the current performance data of the second hardware platform can be determined to be valid and the final test pass result can be generated. If the score is too low, it can be determined that the second hardware platform is abnormal.

[0047] Therefore, by collecting multi-channel data generated during the operation of the first hardware platform in a real online environment, and constructing a scenario description script based on this data that reflects runtime sequence and environmental characteristics, the online runtime scenario can be uniformly replayed in an offline environment. This allows for the reproduction of timing and concurrency in the real online environment in an offline setting, enabling the second hardware platform under test to execute the corresponding runtime process under controlled conditions. By performing consistency calculations on the runtime results obtained by the first and second hardware platforms in the same scenario, the differences between the two runtime results can be quantitatively evaluated. This allows for repeatable testing and evaluation of the second hardware platform's performance, reducing the risk of test results being affected by environmental fluctuations and improving the accuracy and reliability of hardware platform performance evaluation.

[0048] Figure 2 This is a schematic diagram illustrating a scene description script according to an exemplary embodiment.

[0049] like Figure 2 As shown, the scene description script 200 includes three parts: first description information 201, second description information 202, and third description information 203. The first description information 201 includes a time axis “time_axis” and an arrival model “arrival_model”, defining the playback time and triggering logic. The second description information 202 includes a data channel type “ch_type” and the amount of data under that channel “events_ref”. The third description information 203 includes an environmental disturbance parameter “latency_ms” for the network “network”, used to simulate interference factors in the online environment.

[0050] Understandable, Figure 2 The following is an illustrative example using only a portion of the scene description script; however, the embodiments disclosed herein are not limited thereto, and the data in the scene description script is not limited.

[0051] In some embodiments, collecting multi-channel data from a first hardware platform running in an online environment includes: performing time alignment processing on the first channel data, the second channel data, and the third channel data based on their respective timestamps.

[0052] In the example, since the timestamps of the data collected from each channel are different, time-series alignment of the multi-channel data is a prerequisite for reproducing the online environment. This ensures that the correct time relationship is established between the data from the three channels at a given moment. In the embodiments of this disclosure, the data from the first channel, the second channel, and the third channel can be sorted according to their respective timestamps, and a unified time reference can be established based on the timestamps, so that data from different channels can be mapped to the same timeline.

[0053] Therefore, by performing time alignment processing on multi-channel data, the deviation in acquisition time of different data sources can be eliminated, and the data of each channel can be mapped on the same time axis, thus providing a basis for accurately reflecting the multi-channel data correlation relationship during the operation of the first hardware platform.

[0054] In some embodiments, loading the scene description script onto the second hardware platform to be tested for offline playback testing includes: inputting data of each channel in the first sub-multi-channel data segment into the second hardware platform based on the first description information and the second description information; and adding external interference factors to the second hardware platform based on the third description information.

[0055] In the example, during offline testing on the second hardware platform, timing control is applied to the first sub-multi-channel data segment based on the input times of each channel's data as described in the first description information. Furthermore, according to the input data volume at the corresponding time as described in the second description information, the data from each channel is sequentially input to the second hardware platform. For example, based on the time set in the first description information and according to the data volume defined in the second description information, the data from each channel in the first sub-multi-channel data segment is injected into the second hardware platform in a sequential manner. Simultaneously, during the data input process, corresponding interference conditions are introduced into the operating environment of the second hardware platform based on the description of external influencing factors in the third description information. For example, based on the resource consumption values ​​recorded in the third description information, the underlying interface is called to dynamically apply corresponding CPU computing load, network bandwidth latency, and other interference factors to the second hardware platform.

[0056] Therefore, in this way, the second hardware platform can run offline according to the timing and input characteristics corresponding to the online environment, and at the same time reflect the impact of changes in the external environment on the running process, thereby improving the fidelity of offline playback testing and making the testing process closer to the actual running state.

[0057] In some embodiments, a consistency calculation is performed on the first operating result obtained by operating the first hardware platform and the second operating result obtained by operating the second hardware platform to generate a consistency evaluation score. This includes: calculating a distribution consistency score, a time-series consistency score, an event location consistency score, and a service output consistency score between the first and second operating results. The distribution consistency score is used to characterize the degree of matching between the data distribution in the offline playback test environment and the data distribution in the online environment. The time-series consistency score is used to verify the accuracy of the amount of data output by each channel at a specific time. The event location consistency score is used to characterize the degree of temporal consistency between the data output by each channel in the offline playback test environment and the online environment. The service output consistency score is used to verify the accuracy of the calculation results output by the second hardware platform in the offline playback test environment. The distribution consistency score, time-series consistency score, event location consistency score, and service output consistency score are weighted and summed according to a preset weight ratio to generate a consistency evaluation score.

[0058] In the example, the running result can be a data metric generated by the hardware platform during runtime to characterize its performance, including latency distribution, jitter tail latency, timeout error rate, and abnormal event location during the running process. The consistency calculation can be a statistical calculation of these data metrics, that is, calculating four sub-consistency scores of the first running result and the second running result from four dimensions. For distribution consistency scores, the Kolmogorov-Smirnov distance (KS distance) can be used to compare the similarity between the offline replay test and the online real-world data distribution, verifying whether the offline replay test environment replicates the data distribution of the online environment. For time series consistency scores, a dynamic time warping algorithm can be introduced to calculate the shortest path distance between two time series, verifying the accuracy of the output data of each channel in the time dimension. Furthermore, by comparing the occurrence positions of key data or events in each channel on the timeline, the event position consistency score can be obtained by calculating the error of the event position, characterizing the degree of temporal consistency between the offline replay test environment and the online environment. Finally, the classification consistency rate is calculated and compared with the calculation results output in the offline replay test environment and the calculation results output in the online environment to obtain the business output consistency score, verifying the accuracy of the output results of the second hardware platform.

[0059] In the example, after calculating the consistency scores for the four sub-dimensions mentioned above, the distribution consistency score, time-series consistency score, event location consistency score, and business output consistency score can be weighted and summed according to preset weight ratios to generate the final consistency evaluation score. Furthermore, weights can be allocated based on the focus of the test scenario; for example, increasing the weight of business output consistency in functional verification scenarios and increasing the weight of time-series consistency in stress testing scenarios.

[0060] Therefore, by performing consistency calculations from multiple dimensions, the performance of the second hardware platform in offline replay testing can be comprehensively evaluated in a quantitative form, thereby improving the credibility of the test results.

[0061] Figure 3 This is a schematic diagram illustrating a consistency score weighted sum according to an exemplary embodiment.

[0062] like Figure 3 As shown, the weights of the distribution consistency score 301 (w1), the time sequence consistency score 302 (w2), the event location consistency score 303 (w3), and the business output consistency score 304 (w4) can be set respectively. After multiplying by their respective weights, the results are summed to generate the final consistency evaluation score 305.

[0063] In some embodiments, the performance of the second hardware platform is evaluated based on the consistency evaluation score to obtain test results, including: determining that the performance of the second hardware platform is qualified in response to the consistency evaluation score being greater than or equal to a first preset threshold; performing supplementary testing on the second hardware platform in response to the consistency evaluation score being less than the first preset threshold but greater than or equal to a second preset threshold; and determining that the performance of the second hardware platform is unqualified in response to the consistency evaluation score being less than the second preset threshold.

[0064] In the example, the first preset threshold is an upper limit reference value used to determine whether the performance of the second hardware platform meets the expected requirements, and the second preset threshold is a lower limit reference value used to distinguish between acceptable and unacceptable performance ranges. The first preset threshold is greater than the second preset threshold. These two preset thresholds can be set according to specific testing requirements or evaluation criteria, and are used to grade the consistency evaluation score.

[0065] In the example, the calculated consistency evaluation score can be compared with two static thresholds. For example, the first preset threshold can be set to 80 points and the second preset threshold to 60 points. When the consistency evaluation score is greater than or equal to the first preset threshold, for example, when the consistency evaluation score is 82 points, the performance of the second hardware platform can be determined to be qualified. When the consistency evaluation score is less than the first preset threshold but greater than or equal to the second preset threshold, for example, when the consistency score is 75 points, the performance of the second hardware platform can be determined to be in an intermediate state, and supplementary testing of the second hardware platform can be performed. When the consistency evaluation score is less than the second preset threshold, for example, when the consistency score is 54 points, the performance of the second hardware platform can be determined to be unqualified.

[0066] Therefore, by introducing a first and second dual threshold and establishing a supplementary testing mechanism, the performance of the second hardware platform can be graded based on a quantified consistency evaluation score, avoiding subjective judgment and improving the accuracy of performance evaluation results.

[0067] In some embodiments, the online-offline based hardware platform performance testing method further includes: in response to a test result indicating supplementary testing or failure, identifying a second sub-multichannel data segment in which the second hardware platform exhibits a consistency deviation during offline playback testing; and analyzing and processing the data of each channel in the second sub-multichannel data segment to generate an evaluation report for auditing.

[0068] In the example, when the test results indicate that supplementary testing is required or the test is unsatisfactory, the consistency score time-series curve during the offline playback process can be traced back to pinpoint the specific time range within which the score is below the acceptable threshold. The corresponding multi-channel data, i.e., the second sub-multi-channel data segment, is then extracted from this time range. Subsequently, the data from each channel within the second sub-multi-channel data segment is analyzed and processed. The performance of different channels during this time period is organized and summarized. For example, if a consistency deviation occurs due to excessive business request volume at a certain moment, it may be due to an anomaly in a computing module of the second hardware platform. Finally, the analysis results are summarized to generate an evaluation report for auditing purposes.

[0069] Therefore, by processing the test results, data segments that show consistency deviations in offline playback tests can be analyzed, thereby providing traceable data for subsequent problem localization and auditing.

[0070] Embodiments of this disclosure also provide an online-offline hardware platform performance testing device.

[0071] Figure 4 This is a schematic block diagram illustrating an online-offline based hardware platform performance testing apparatus 400 according to an exemplary embodiment.

[0072] In some embodiments, the online-offline based hardware platform performance testing device 400 may include a data acquisition module 401, a data recognition module 402, a script generation module 403, a script loading module 404, a score generation module 405, and a result determination module 406.

[0073] The data acquisition module 401 is configured to acquire multi-channel data from a first hardware platform running in an online environment. The multi-channel data is associated with multiple channels and includes first-channel data for recording the first hardware platform's processing of business requests, second-channel data acquired from sensors, and third-channel data reflecting the hardware resource consumption status.

[0074] The data recognition module 402 is configured to perform abnormal data recognition processing on multi-channel data to obtain a first sub-multi-channel data segment including abnormal data.

[0075] The script generation module 403 is configured to generate a scenario description script based on the data and timestamp of each channel in the first sub-multi-channel data segment. The scenario description script includes first description information, second description information and third description information. The first description information is used to describe the first moment when the data of each channel in the first sub-multi-channel data segment is input. The second description information is used to describe the amount of input data of each channel in the first sub-multi-channel data segment at the first moment. The third description information is used to describe external interference factors in the online environment.

[0076] The script loading module 404 is configured to load the scene description script to the second hardware platform to be tested for offline playback testing. The second hardware platform includes a playback engine, which is configured to parse the scene description script and run the parsed scene description script.

[0077] The score generation module 405 is configured to perform a consistency calculation on the first running result obtained by operating the first hardware platform and the second running result obtained by operating the second hardware platform to generate a consistency evaluation score.

[0078] The results determination module 406 is configured to evaluate the performance of the second hardware platform based on the conformance evaluation score to obtain test results.

[0079] It should be understood that Figure 4 The data acquisition module 401, data recognition module 402, script generation module 403, script loading module 404, score generation module 405, and result determination module 406 shown can respectively correspond to, for example, the following modules: Figure 1 The operations of steps S101, S102, S103, S104, S105, and S106 are shown. Therefore, the details of each aspect will not be elaborated here.

[0080] In some embodiments, the data acquisition module 401 may include a time alignment module 4011. The time alignment module 4011 may be configured to perform time alignment processing on the first channel data, the second channel data, and the third channel data based on their respective timestamps.

[0081] In some embodiments, the script loading module 404 may include a data input module 4041 and an interference addition module 4042. The data input module 4041 may be configured to input data from each channel of the first sub-multichannel data segment into the second hardware platform based on first and second description information. The interference addition module 4042 may be configured to add external interference factors to the second hardware platform based on third description information.

[0082] In some embodiments, the score generation module 405 may include a consistency calculation module 4051 and a score weighting module 4052. The consistency calculation module 4051 may be configured to calculate a distribution consistency score, a time-series consistency score, an event location consistency score, and a service output consistency score between the first and second running results. The distribution consistency score characterizes the degree of matching between the data distribution in the offline playback test environment and the data distribution in the online environment; the time-series consistency score verifies the accuracy of the amount of data output by each channel at a specific time; the event location consistency score characterizes the temporal consistency of the data output by each channel in the offline playback test environment and the online environment; and the service output consistency score verifies the accuracy of the calculation results output by the second hardware platform in the offline playback test environment. The score weighting module 4052 may be configured to weight and sum the distribution consistency score, time-series consistency score, event location consistency score, and service output consistency score according to a preset weight ratio to generate a consistency evaluation score.

[0083] In some embodiments, the result determination module 406 may include a first result determination module 4061, a second result determination module 4062, and a third result determination module 4063. The first result determination module 4061 may be configured to determine that the performance of the second hardware platform is qualified in response to a consistency evaluation score greater than or equal to a first preset threshold. The second result determination module 4062 may be configured to perform supplementary testing on the second hardware platform in response to a consistency evaluation score less than the first preset threshold but greater than or equal to a second preset threshold. The third result determination module 4063 may be configured to determine that the performance of the second hardware platform is unqualified in response to a consistency evaluation score less than the second preset threshold.

[0084] In some embodiments, the online-offline based hardware platform performance testing apparatus 400 may further include a deviation determination module 407 and a report generation module 408. The deviation determination module 407 may be configured to determine a second sub-multichannel data segment in which a consistency deviation occurs during offline playback testing of the second hardware platform, in response to a test result indicating supplementary testing or failure. The report generation module 408 may be configured to analyze and process the data of each channel in the second sub-multichannel data segment to generate an evaluation report for auditing purposes.

[0085] According to one aspect of this disclosure, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0086] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0087] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0088] Figure 5 An example computer device 500 is shown in which any of the embodiments described herein may be implemented. The computer device 500 may be used to implement one or more components of the systems and methods described above. The computer device 500 may include a bus 502 or other communication mechanism for communicating information, and one or more processors 504 coupled to the bus 502 for processing information. The processor 504 may be, for example, one or more general-purpose microprocessors.

[0089] Computer device 500 may also include main memory 506, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 502, for storing information and instructions to be executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 504. Such instructions, when stored in a storage medium accessible to processor 504, can make computer device 500 a special-purpose machine customized to perform the operations specified in the instructions. Main memory 506 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common media formats may include, for example, floppy disks, collapsible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs (read-only optical disc drives), any other optical data storage media, any physical media with a perforated arrangement, RAM (random access memory), DRAM (dynamic random access memory), PROM (programmable read-only memory) and EPROM (erasable programmable read-only memory), FLASH-EPROM (fast erase programmable read-only memory), NVRAM (non-volatile random access memory), any other memory chips or tape cartridges, or network versions of the above.

[0090] Computer device 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs (Application-Specific Integrated Circuits) or FPGAs (Field-Programmable Gate Arrays), firmware, and / or program logic, which, when combined with computer device 500, enable computer device 500 to become a special-purpose machine or to be programmed therein. According to one embodiment, the techniques described herein are executed by computer device 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 508. Executing the sequence of instructions contained in main memory 506 causes processor 504 to perform the processing steps described herein. For example, the processes / methods disclosed herein may be implemented by computer program instructions stored in main memory 506. When these instructions are executed by processor 504, they may perform the steps shown in the corresponding figures and as described above. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0091] Computer device 500 also includes a network interface 510 coupled to bus 502. Network interface 510 can provide bidirectional data communication coupled to one or more network links connected to one or more networks. As another example, network interface 510 can be a local area network (LAN) card to provide data communication connectivity with a compatible LAN (or a WAN component communicating with a WAN (wide area network)). Wireless links can also be implemented.

[0092] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0093] Each process, method, and algorithm described in the preceding sections can be embodied in a code module executed by one or more computer systems or computer processors including computer hardware, and can be fully or partially automated by them. These processes and algorithms can be implemented, in part or in whole, in a specific application circuit.

[0094] When the functions disclosed herein are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, can be embodied in the form of a software product. This software product can be stored in a storage medium and includes instructions to cause a computer device (which may be a personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the embodiments of this application. The storage medium may include a flash drive, a portable hard drive, ROM, RAM, a magnetic disk, an optical disk, another medium suitable for storing program code, or any combination thereof.

[0095] The embodiments disclosed herein can be implemented via a cloud platform, server, or group of servers that interact with a client. The client can be a terminal device or a client registered by a user on the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can install platform applications.

[0096] The various features and processes described above can be used independently or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Furthermore, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in a non-specifically disclosed order, or multiple blocks or states may be combined in a single block or state. Exemplary blocks or states may be executed serially, in parallel, or otherwise. Blocks or states may be added to or removed from the disclosed exemplary embodiments. The exemplary systems and components described herein may be configured differently from those described. For example, elements may be added, removed, or rearranged compared to the disclosed exemplary embodiments.

[0097] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. An algorithm may consist of program code or instructions stored in memory (such as the non-transitory computer-readable storage medium described above). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not be explicitly programmed into the computer to perform the function, but may learn from training data to obtain a predictive model for performing that function.

[0098] The various operations of the exemplary methods described herein can be performed at least in part by one or more processors, which are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute the engine of a processor implementation whose operation is to perform one or more of the operations or functions described herein.

[0099] Similarly, the methods described herein can be implemented at least partially by a processor, where a specific processor or one or more processors are examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors can also run in a “cloud computing” environment or as “Software as a Service” (SaaS) to support the execution of the relevant operations. For example, at least some operations can be performed by a group of computers (as an example of a machine including processors), which can be accessed via a network (e.g., the Internet) and through one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0100] The performance of certain operations can be distributed across processors, not just residing within a single machine, but deployed across many machines. In some exemplary embodiments, the processor or the processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other exemplary embodiments, the processor or the processor-implemented engine may be distributed across many geographic locations.

[0101] In this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are described and illustrated as independent operations, one or more individual operations may be performed concurrently, and these operations are not required to be performed in the order shown. Structures and functionalities presented as independent components in the example configuration may be implemented as combined structures or components. Similarly, structures and functionalities presented as individual components may be implemented as independent components. These and other variations, modifications, additions, and improvements are all within the scope of this document.

[0102] As used herein, “or” is inclusive rather than exclusive unless explicitly stated or indicated by context. Furthermore, “and” is both common and individual unless explicitly stated or indicated by context. Moreover, multiple instances may be provided for the resources, operations, or structures described herein as a single example. Furthermore, the boundaries between various resources, operations, engines, and data stores are somewhat arbitrary, and specific operations are illustrated within the context of a particular illustrative configuration. The allocation of other functionalities is conceivable and may fall within the scope of various embodiments of this disclosure. Generally, structures and functionalities presented as independent resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functionalities presented as individual resources may be implemented as independent resources. These and other variations, modifications, additions, and improvements are all within the scope of embodiments of this disclosure. Therefore, this specification and accompanying drawings should be viewed in an illustrative rather than restrictive sense.

[0103] The terms “comprising” or “including” are used to indicate the presence of a subsequently stated feature, but do not preclude the addition of other features. Conditional language, in particular, such as “may,” “can,” or “may,” unless specifically stated or otherwise understood in the context of use, is generally intended to express that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Therefore, such conditional language generally does not imply that a feature, element, and / or step is necessary in any way for one or more embodiments, or that one or more embodiments must include logic that, with or without user input or prompting, determines whether such features, elements, and / or steps are included in any particular embodiment, or whether they are to be performed in any particular embodiment.

Claims

1. A hardware platform performance testing method based on online-offline testing, characterized in that, The method includes: Collect multi-channel data from a first hardware platform running in an online environment. The multi-channel data is associated with multiple channels and includes first-channel data for recording the first hardware platform's processing of business requests, second-channel data acquired from sensors, and third-channel data reflecting the hardware resource consumption status. The multi-channel data is subjected to anomaly identification processing to obtain a first sub-multi-channel data segment including the anomaly data; Based on the data and timestamp of each channel in the first sub-multi-channel data segment, a scenario description script is generated. The scenario description script includes first description information, second description information and third description information. The first description information is used to describe the first moment when the data of each channel in the first sub-multi-channel data segment is input. The second description information is used to describe the amount of input data of each channel in the first sub-multi-channel data segment at the first moment. The third description information is used to describe the external interference factors in the online environment. The scene description script is loaded onto the second hardware platform to be tested for offline playback testing. The second hardware platform includes a playback engine, which is configured to parse the scene description script and run the parsed scene description script. A consistency calculation is performed on the first operating result obtained from operating the first hardware platform and the second operating result obtained from operating the second hardware platform to generate a consistency evaluation score. This consistency calculation includes: Calculate the distribution consistency score, time series consistency score, event location consistency score, and service output consistency score between the first running result and the second running result. The distribution consistency score is used to characterize the degree of matching between the data distribution in the offline playback test environment and the data distribution in the online environment. The time series consistency score is used to verify the accuracy of the amount of data output by each channel at a specific time. The event location consistency score is used to characterize the degree of consistency between the data output by each channel in the offline playback test environment and the online environment in terms of time. The service output consistency score is used to verify the accuracy of the calculation results output by the second hardware platform in the offline playback test environment. The distribution consistency score, the time series consistency score, the event location consistency score, and the service output consistency score are weighted and summed according to preset weight ratios to generate the consistency evaluation score; and The performance of the second hardware platform is evaluated based on the consistency evaluation score to obtain test results.

2. The method according to claim 1, characterized in that, The collection of multi-channel data from the first hardware platform running in the online environment includes: Based on the timestamps of the first channel data, the second channel data, and the third channel data, time alignment processing is performed on the first channel data, the second channel data, and the third channel data.

3. The method according to claim 1 or 2, characterized in that, The step of loading the scene description script onto the second hardware platform to be tested for offline playback testing includes: Based on the first description information and the second description information, data from each channel of the first sub-multi-channel data segment is input to the second hardware platform; and Based on the third description information, the external interference factor is added to the second hardware platform.

4. The method according to claim 1, characterized in that, The evaluation of the performance of the second hardware platform based on the consistency evaluation score to obtain test results includes: If the consistency evaluation score is greater than or equal to a first preset threshold, the performance of the second hardware platform is determined to be qualified. In response to the consistency evaluation score being less than the first preset threshold and greater than or equal to the second preset threshold, supplementary testing is performed on the second hardware platform; and If the consistency evaluation score is less than the second preset threshold, the performance of the second hardware platform is determined to be unqualified.

5. The method according to claim 4, characterized in that, The method further includes: In response to the test result being either supplementary testing or failure, a second sub-multichannel data segment with a consistency deviation is identified as occurring on the second hardware platform during the offline playback test; and The data from each channel in the second sub-multi-channel data segment are analyzed and processed to generate an evaluation report for auditing.

6. A hardware platform performance testing device based on online-offline testing, characterized in that, The device includes: The data acquisition module is configured to acquire multi-channel data from a first hardware platform running in an online environment. The multi-channel data is associated with multiple channels and includes first-channel data for recording the first hardware platform's processing of business requests, second-channel data acquired from sensors, and third-channel data reflecting the hardware resource consumption status. The data identification module is configured to perform abnormal data identification processing on the multi-channel data to obtain a first sub-multi-channel data segment including abnormal data. The script generation module is configured to generate a scenario description script based on the data and timestamp of each channel in the first sub-multi-channel data segment. The scenario description script includes first description information, second description information, and third description information. The first description information is used to describe the first moment when the data of each channel in the first sub-multi-channel data segment is input. The second description information is used to describe the amount of input data of each channel in the first sub-multi-channel data segment at the first moment. The third description information is used to describe the external interference factors in the online environment. The script loading module is configured to load the scene description script onto the second hardware platform to be tested for offline playback testing. The second hardware platform includes a playback engine, which is configured to parse the scene description script and run the parsed scene description script. The score generation module is configured to perform a consistency calculation on the first running result obtained from operating the first hardware platform and the second running result obtained from operating the second hardware platform to generate a consistency evaluation score. The score generation module includes a consistency calculation module and a score weighting module. The consistency calculation module is configured to calculate the distribution consistency score, time sequence consistency score, event location consistency score, and service output consistency score between the first running result and the second running result. The distribution consistency score is used to characterize the degree of matching between the data distribution in the offline playback test environment and the data distribution in the online environment. The time sequence consistency score is used to verify the accuracy of the amount of data output by each channel at a specific time. The event location consistency score is used to characterize the degree of temporal consistency between the data output by each channel in the offline playback test environment and the online environment. The service output consistency score is used to verify the accuracy of the calculation results output by the second hardware platform in the offline playback test environment. The score weighting module is configured to perform a weighted summation of the distribution consistency score, the time series consistency score, the event location consistency score, and the service output consistency score according to a preset weight ratio to generate the consistency evaluation score; and The result determination module is configured to evaluate the performance of the second hardware platform based on the consistency evaluation score to obtain test results.

7. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-5.

Images