Text interaction traffic robustness automatic testing method and device

Abstract

The application provides a text interaction traffic robustness automatic testing method and device, the method comprises the following steps: based on the test target, different scenes are configured, different scene configurations are combined to obtain test cases covering different scenes, multiple parameters of the test cases are configured, the test cases are grouped and prioritized based on the test requirements, the configuration parameters required for testing are processed to obtain configuration parameters that meet the parameter receiving requirements of the large model firewall system, the configuration parameters are sent to the large model firewall system, the large model firewall system is preconfigured based on the configuration parameters, simulated test traffic is sent to the large model firewall system based on the test case parameters for traffic detection, the traffic detection results returned by the large model firewall system are received, the target test data of the whole test process is collected and processed, the core evaluation indexes are calculated based on the processed target test data, and the visual test report is generated based on the core evaluation indexes.

Description

Technical Field

[0001] This application relates to the field of large model testing technology, and more specifically, to an automated testing method and apparatus for the robustness of text interaction traffic. Background Technology

[0002] When enterprises apply large AI models, they often face security threats. These AI risks pose significant risks to businesses. The Large AI Model Firewall (LLM Firewall) is a security protection system specifically designed for these large language models. It monitors user input and model output in real time to prevent keyword injection, jailbreaking, and leakage of sensitive information.

[0003] Currently, as testing progresses, robustness testing for AI large-scale firewall detection is particularly important. This needs to be performed through manual testing, that is, by using personal experience to conduct robustness testing on the dialogue interaction of AI large-scale firewalls.

[0004] However, current manual testing methods cover a limited amount of text, resulting in a narrow range of vocabulary and scenarios covered, which means the testing results are generally mediocre. Furthermore, manual testing relies heavily on personal experience and is subject to randomness, reducing testing efficiency and increasing testing and maintenance costs. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide an automated method and apparatus for robustness testing of text interaction traffic. This method generates test cases covering different scenarios and configures various parameters for these test cases. It pre-configures a large-scale firewall system using these parameters and sends simulated test traffic to the large-scale firewall system according to the test case parameters for traffic detection. Finally, it calculates core evaluation indicators based on the data from the entire testing process and generates a visual test report. This method comprehensively covers different scenarios, increases the number of words and characters covered, and enables comprehensive verification of massive attack scenarios, avoiding blind spots in firewall defense. Furthermore, it eliminates reliance on manual intervention, improves testing effectiveness and efficiency, and reduces testing and maintenance costs.

[0006] In a first aspect, embodiments of this application provide an automated testing method for the robustness of text interaction traffic, the method comprising: Based on the test objectives, different scenarios are configured, and the configurations of different scenarios are combined to obtain test cases that cover different scenarios; For the test cases, configure various test case parameters, group the test cases based on test requirements, and prioritize the test cases. The configuration parameters required for testing are processed to obtain configuration parameters that meet the parameter receiving requirements of the large-scale firewall system, and the configuration parameters are sent to the large-scale firewall system. The large-scale firewall system is then pre-configured based on the configuration parameters. Based on the test case parameters, simulated test traffic is sent to the large-scale firewall system for traffic detection, and the traffic detection results returned by the large-scale firewall system are received; wherein, the simulated test traffic represents the simulation of real business traffic; Collect target test data throughout the entire testing process, perform multi-dimensional processing on the target test data, calculate the corresponding core evaluation indicators based on the processed target test data, and generate the corresponding visual test report based on the core evaluation indicators.

[0007] In one possible implementation, the different configuration scenarios include: The system randomly generates multiple words covering different scenarios from a preset word library, scrambles the words, and then splices them together; it also randomly extracts multiple Chinese characters from a preset Chinese character library, scrambles the characters, and then splices them together. Configure punctuation marks from a preset punctuation mark library, configure Chinese special characters from a preset special character library, and randomly generate multiple words from a preset word library.

[0008] In one possible implementation, combining configurations for different scenarios to obtain test cases covering different scenarios includes: The concatenated words, Chinese characters, punctuation marks, and special Chinese characters are randomly combined to generate multiple original elements. The original elements are then shuffled and concatenated to obtain the corresponding first test case and second test case. The concatenated words, Chinese characters, punctuation marks, Chinese special characters, and words are randomly combined to generate multiple original elements. These original elements are then shuffled and concatenated to obtain the corresponding third test case. For any test case, multiple original elements are generated, the original elements are shuffled and then spliced ​​together, and the splicing result is randomly encoded for testing to obtain the corresponding fourth test case; wherein, the fourth test case represents the escape scenario under the protection capability.

[0009] In one possible implementation, after the configuration parameters are sent to the large-scale firewall system, the method further includes: The actual configuration of the large-scale firewall system is verified based on the configuration parameters, and the consistency between the configuration parameters and the actual configuration parameters of the large-scale firewall system is compared. If the configuration parameters and the actual configuration parameters are inconsistent, the configuration parameters will be automatically resent.

[0010] In one possible implementation, sending the simulated test traffic to the large-scale firewall system includes: Adjust the format of the generated simulated test traffic to match that of real business traffic and adapt it to the traffic inspection standards of the large model firewall system. Based on the test case parameters configured in the test case, the corresponding traffic parameters are determined, and the simulated test traffic is sent to the large model firewall system based on the traffic parameters, while the traffic sending status is monitored in real time during the sending process. If the traffic transmission status is abnormal, the simulated test traffic will be retransmitted and the abnormal information will be fed back.

[0011] In one possible implementation, the method further includes: Real-time monitoring of key indicators throughout the entire process of simulated test traffic transmission; wherein, the key indicators include at least transmission rate, number of transmissions, and percentage of successful transmissions; The traffic parameters are dynamically adjusted based on the key indicators, and the simulated test traffic is sent to the large-scale firewall system based on the adjusted traffic parameters.

[0012] In one possible implementation, the method further includes: Based on the aforementioned test objectives, select the corresponding configuration range; Based on the configuration range, the corresponding configuration parameters are selected as the configuration parameters required for testing and sent to the large-scale firewall system.

[0013] Secondly, embodiments of this application also provide an automated testing device for the robustness of text interaction traffic, the device comprising: The acquisition module is used to configure different scenarios based on the test objectives, and to combine the configurations of different scenarios to obtain test cases covering different scenarios; The configuration module is used to configure various test case parameters for the test cases, group the test cases based on test requirements, and prioritize the test cases. The distribution module is used to process the configuration parameters required for testing to obtain configuration parameters that meet the parameter receiving requirements of the large-scale firewall system, and distribute the configuration parameters to the large-scale firewall system to pre-configure the large-scale firewall system based on the configuration parameters. The detection module is used to send simulated test traffic to the large-scale firewall system for traffic detection based on the test case parameters, and to receive the traffic detection results returned by the large-scale firewall system; wherein, the simulated test traffic represents the simulation of real business traffic; The generation module is used to collect target test data throughout the entire testing process, perform multi-dimensional processing on the target test data, calculate the corresponding core evaluation indicators based on the processed target test data, and generate the corresponding visual test report based on the core evaluation indicators.

[0014] In one possible implementation, the acquisition module is specifically used for: The system randomly generates multiple words covering different scenarios from a preset word library, scrambles the words, and then splices them together; it also randomly extracts multiple Chinese characters from a preset Chinese character library, scrambles the characters, and then splices them together. Configure punctuation marks from a preset punctuation mark library, configure Chinese special characters from a preset special character library, and randomly generate multiple words from a preset word library.

[0015] In one possible implementation, the acquisition module is specifically used for: The concatenated words, Chinese characters, punctuation marks, and special Chinese characters are randomly combined to generate multiple original elements. The original elements are then shuffled and concatenated to obtain the corresponding first test case and second test case. The concatenated words, Chinese characters, punctuation marks, Chinese special characters, and words are randomly combined to generate multiple original elements. These original elements are then shuffled and concatenated to obtain the corresponding third test case. For any test case, multiple original elements are generated, the original elements are shuffled and then spliced ​​together, and the splicing result is randomly encoded for testing to obtain the corresponding fourth test case; wherein, the fourth test case represents the escape scenario under the protection capability.

[0016] In one possible implementation, the device further includes: The verification module is used to verify the actual configuration of the large-scale firewall system based on the configuration parameters after the configuration parameters are sent to the large-scale firewall system, and to compare the consistency between the configuration parameters and the actual configuration parameters of the large-scale firewall system. The retransmission module is used to automatically retransmit the configuration parameters when the configuration parameters and the actual configuration parameters are inconsistent.

[0017] In one possible implementation, the detection module is specifically used for: Adjust the format of the generated simulated test traffic to match that of real business traffic and adapt it to the traffic inspection standards of the large model firewall system. Based on the test case parameters configured in the test case, the corresponding traffic parameters are determined, and the simulated test traffic is sent to the large model firewall system based on the traffic parameters, while the traffic sending status is monitored in real time during the sending process. If the traffic transmission status is abnormal, the simulated test traffic will be retransmitted and the abnormal information will be fed back.

[0018] In one possible implementation, the device further includes: The monitoring module is used to monitor key indicators of the entire process of the simulated test traffic transmission in real time; wherein, the key indicators include at least the transmission rate, the number of transmissions, and the percentage of successful transmissions. The adjustment module is used to dynamically adjust the traffic parameters based on the key indicators, and send the simulated test traffic to the large model firewall system based on the adjusted traffic parameters.

[0019] In one possible implementation, the device further includes: The first selection module is used to select the corresponding configuration range based on the test objective; The second selection module is used to select the corresponding configuration parameters based on the configuration range as the configuration parameters required for testing and send them to the large-scale firewall system.

[0020] Thirdly, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the automated text interaction traffic robustness testing method as described in any of the first aspects.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, performs the steps of the automated text interaction traffic robustness testing method described in any one of the first aspects.

[0022] This application provides an automated method and apparatus for robustness testing of text interaction traffic. It configures different scenarios based on the test objective, combines the configurations of different scenarios to obtain test cases covering different scenarios, configures various test case parameters for each test case, groups the test cases based on test requirements, prioritizes the test cases, processes the configuration parameters required for testing to obtain configuration parameters that meet the parameter receiving requirements of a large-scale firewall system, and sends the configuration parameters to the large-scale firewall system. Based on the configuration parameters, the large-scale firewall system is pre-configured. Based on the test case parameters, simulated test traffic is sent to the large-scale firewall system for traffic detection, and the traffic detection results returned by the large-scale firewall system are received. Target test data is collected throughout the entire testing process, and the target test data is processed in multiple dimensions. Based on the processed target test data, corresponding core evaluation indicators are calculated, and a corresponding visual test report is generated based on the core evaluation indicators. This application generates test cases covering different scenarios and configures various parameters for these test cases. It pre-configures a large-scale firewall system using these parameters and sends simulated test traffic to the firewall system according to the test case parameters for traffic detection. Finally, it calculates core evaluation metrics based on the data from the entire testing process and generates a visual test report. This comprehensive approach covers diverse scenarios, increases the number of words and characters covered, and enables full verification of massive attack scenarios, avoiding blind spots in firewall defense. Furthermore, it eliminates reliance on manual intervention, improving testing effectiveness and efficiency while reducing testing and maintenance costs.

[0023] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of an automated text interaction traffic robustness testing method provided according to an embodiment of this application; Figure 2 This is a schematic diagram of the overall process of automated testing for the robustness of text interaction traffic; Figure 3 This is a schematic diagram of the structure of the automated text interaction traffic robustness testing device provided in the embodiments of this application; Figure 4This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0027] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0028] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0029] Given the security threats often faced when enterprises apply large AI models, these AI risks pose significant risks to businesses. The AI ​​Large Model Firewall (LLM Firewall) is a security protection system specifically designed for these large language models. It monitors user input and model output in real time to prevent keyword injection, jailbreaking, and leakage of sensitive information.

[0030] Currently, as testing progresses, robustness testing for AI large-scale firewall detection is particularly important. This needs to be performed through manual testing, that is, by using personal experience to conduct robustness testing on the dialogue interaction of AI large-scale firewalls.

[0031] However, current manual testing methods cover a limited amount of text, resulting in a narrow range of vocabulary and scenarios covered, which means the testing results are generally mediocre. Furthermore, manual testing relies heavily on personal experience and is subject to randomness, reducing testing efficiency and increasing testing and maintenance costs.

[0032] To address this issue, this application provides an automated method and apparatus for robustness testing of text interaction traffic. It generates test cases covering different scenarios and configures various parameters for these test cases. The method pre-configures a large-scale firewall system using these parameters and sends simulated test traffic to the firewall system according to the test case parameters for traffic detection. Finally, it calculates core evaluation indicators based on the data from the entire testing process and generates a visual test report. This approach comprehensively covers different scenarios, increases the number of words and characters covered, and enables comprehensive verification of massive attack scenarios, avoiding blind spots in firewall defense. Furthermore, it eliminates reliance on manual intervention, improving testing effectiveness and efficiency while reducing testing and maintenance costs.

[0033] Figure 1 This is a flowchart of an automated text interaction traffic robustness testing method provided according to an embodiment of this application. For example... Figure 1 As shown in the embodiments of this application, the automated testing method for text interaction traffic robustness may specifically include: S101. Configure different scenarios based on the test objectives, and combine the configurations of different scenarios to obtain test cases covering different scenarios.

[0034] S102. Configure various test case parameters for test cases, group test cases based on test requirements, and prioritize test cases.

[0035] S103. Process the configuration parameters required for the test to obtain configuration parameters that meet the parameter receiving requirements of the large-scale firewall system, and send the configuration parameters to the large-scale firewall system. Based on the configuration parameters, pre-configure the large-scale firewall system.

[0036] S104. Based on the test case parameters, send simulated test traffic to the large-scale firewall system for traffic detection, and receive the traffic detection results returned by the large-scale firewall system.

[0037] S105. Collect target test data throughout the entire testing process, perform multi-dimensional processing on the target test data, calculate the corresponding core evaluation indicators based on the processed target test data, and generate the corresponding visual test report based on the core evaluation indicators.

[0038] The aforementioned automated robustness testing method for text interaction traffic generates test cases covering different scenarios and configures various parameters for these test cases. It pre-configures a large-scale firewall system using these parameters and sends simulated test traffic to the firewall system according to the test case parameters for traffic detection. Finally, it calculates core evaluation metrics based on data from the entire testing process and generates a visual test report. This method comprehensively covers different scenarios, increases the number of words and characters covered, and enables comprehensive verification of massive attack scenarios, avoiding blind spots in firewall defense. Furthermore, it eliminates reliance on manual intervention, improving testing effectiveness and efficiency while reducing testing and maintenance costs.

[0039] The exemplary steps described above in the embodiments of this application are illustrated below with specific examples: S101 configures different scenarios based on the test objectives, and combines the configurations of different scenarios to obtain test cases covering different scenarios.

[0040] In this embodiment of the application, different scenarios are configured according to the test objective, and the configurations of different scenarios are combined to generate test cases covering different scenarios for subsequent processing. For example, as Figure 2 As shown.

[0041] Optionally, when configuring different scenarios, multiple words covering different scenarios can be randomly generated from a preset word library, shuffled, and then concatenated; multiple Chinese characters can be randomly extracted from a preset Chinese character library, shuffled, and then concatenated; punctuation marks can be configured from a preset punctuation mark library; Chinese special characters can be configured from a preset special character library; and multiple words can be randomly generated from a preset word library. Punctuation marks include sentence-level punctuation and sentence-ending punctuation.

[0042] In general, the configuration of multiple scenarios involves (1) the configuration of the number of words, (2) the configuration of the number of Chinese characters, (3) the configuration of punctuation marks, (4) the configuration of special characters and (5) the configuration of English words. For example, in the configuration of the number of words, 10 random words are generated and then spliced ​​after being shuffled. Other configurations are not described in detail.

[0043] Optionally, when combining configurations for different scenarios to obtain test cases covering different scenarios, the concatenated words, Chinese characters, punctuation marks, and special Chinese characters are randomly combined to generate multiple original elements. These original elements are then shuffled and concatenated to obtain the corresponding first and second test cases. Similarly, the concatenated words, Chinese characters, punctuation marks, special Chinese characters, and words are randomly combined to generate multiple original elements. These original elements are then shuffled and concatenated to obtain the corresponding third test case. For any test case, multiple original elements are generated, shuffled, and concatenated. The concatenated result is then subjected to random encoding testing to obtain the corresponding fourth test case. The fourth test case includes string data and hexadecimal data, and represents an escape scenario under the protection capability.

[0044] In this case, the configurations of (1) the number of words, (2) the number of Chinese characters, (3) the configuration of punctuation marks, and (4) the configuration of special characters are randomly combined (at least one can be selected), and then shuffled and spliced ​​together under different specified lengths to obtain the corresponding splicing results, namely the first test case and the second test case. The second test case represents the test case for the ultra-long scenario. For example, if the number of original elements generated by the random combination of (1) to (4) is 19, and these original elements are shuffled and spliced ​​together, the splicing result is the first test case; if the number of original elements generated by the random combination is 99, and these original elements are shuffled and spliced ​​together, the splicing result is the second test case for the ultra-long scenario.

[0045] Continuing, for example, the configuration of (1) the number of words, (2) the number of Chinese characters, (3) the configuration of punctuation marks, (4) the configuration of special characters, and (5) the configuration of English words are randomly combined to generate 34 original elements. The 34 original elements are then shuffled and concatenated to obtain the concatenated result, which is the third test case. In addition, multiple original elements are generated for any of the above test cases, shuffled, and concatenated. The concatenated result is then randomly encoded and tested to obtain string data and hexadecimal data, which is the fourth test case. Here, it can be understood that the fourth test case includes two test cases: string data and hexadecimal data.

[0046] S102 configures various test case parameters for test cases, groups test cases based on test requirements, and assigns priority to test cases.

[0047] In this embodiment, the test case parameters include at least traffic parameters, triggering conditions, and test rounds. For the test cases obtained above, test case parameters are configured for each test case, and the test cases are grouped and prioritized based on test requirements for subsequent processing. For example, as... Figure 2 As shown.

[0048] Specifically, in the traffic parameter configuration, the sending frequency of test traffic (such as the number of requests sent per second / minute) and the duration of traffic flow can be set. Fixed frequency configuration and dynamic frequency adjustment (such as simulating traffic fluctuations in real-world scenarios) are supported to adapt to traffic testing needs under different business scenarios. In the trigger condition configuration, the trigger thresholds of test cases can be customized (such as specific data volume thresholds, time thresholds, abnormal state thresholds, etc.), trigger delay time and trigger count limits can be set, and the trigger logic for starting and terminating test cases can be clearly defined to avoid invalid test execution. In the test execution configuration, the execution rounds, single round execution duration, and round interval time of test cases can be set. Failure rerun parameter configuration (such as rerun count and rerun interval) is supported. At the same time, the collection frequency and collection dimensions of test results can be configured to ensure the integrity of test data. In test case grouping and priority management, test cases can be customized to group according to test requirements (such as test scenarios, business modules, test types, etc.), and test case priorities can be divided according to the degree of business impact. These priorities can be dynamically adjusted according to test requirements.

[0049] S103 processes the configuration parameters required for testing to obtain configuration parameters that meet the parameter receiving requirements of the large-scale firewall system, and sends the configuration parameters to the large-scale firewall system to pre-configure the large-scale firewall system based on the configuration parameters.

[0050] In this embodiment, the configuration parameters required for testing can be rules, policies, interception thresholds, whitelists, etc. These configuration parameters are processed, and when they meet the parameter receiving requirements of the large-scale firewall system, the configuration parameters are sent to the large-scale firewall system for pre-configuration. For example, ... Figure 2 As shown. In some implementations, after the configuration parameters are sent to the large-scale firewall system, the actual configuration of the large-scale firewall system is verified based on the configuration parameters, and the consistency between the configuration parameters and the actual configuration parameters of the large-scale firewall system is compared; if the configuration parameters and the actual configuration parameters are inconsistent, the configuration parameters are automatically resent.

[0051] It should be noted that when the number of failed attempts to resend configuration parameters exceeds a preset threshold, the test will stop and the corresponding failure reason will be recorded.

[0052] Specifically, the pre-configuration of the large-scale firewall system is completed through the distribution of configuration parameters. This includes enabling rules and policies, setting interception thresholds, and adding whitelists. This involves first standardizing the configuration parameters, performing logical verification and compatibility adjustments to ensure they conform to the large-scale firewall system's parameter receiving specifications. This prevents distribution failures due to incorrect parameter formats or logical conflicts, and ensures the initial state of the large-scale firewall system during testing meets expectations, avoiding test failures due to insufficient pre-configuration. Furthermore, in the pre-configuration check and anomaly handling, the actual configuration of the large-scale firewall system is verified after the configuration parameters are distributed. The consistency between the distributed configuration parameters and the actual effective parameters of the large-scale firewall system is compared. If discrepancies, ineffectiveness, or non-findability occur, the configuration parameters are automatically retransmitted. If retransmission exceeds a threshold (e.g., 3 times) and still fails (e.g., due to network issues), the test is stopped, the scene is preserved, and the reason for failure is recorded for operators to analyze the problem.

[0053] S104, based on the test case parameters, sends simulated test traffic to the large-scale firewall system for traffic detection and receives the traffic detection results returned by the large-scale firewall system.

[0054] In this embodiment, the simulated test traffic represents a simulation of real business traffic; the simulated test traffic is sent to a large-scale firewall system for traffic detection, and the traffic detection results returned by the large-scale firewall system are received. For example, such as... Figure 2 As shown.

[0055] Optionally, when sending simulated test traffic to the large-scale firewall system, the format of the generated simulated test traffic is adjusted to be consistent with real business traffic and adapted to the traffic inspection standards of the large-scale firewall system; the corresponding traffic parameters are determined based on the test case parameters configured in the test case configuration, and the simulated test traffic is sent to the large-scale firewall system based on the traffic parameters, while the traffic sending status is monitored in real time during the sending process; if the traffic sending status is abnormal, the simulated test traffic is resent and the abnormal information is fed back. The traffic parameters include at least: traffic sending frequency, traffic peak value, and duration.

[0056] Specifically, for example, based on traffic parameters (such as traffic transmission frequency, peak traffic, and duration), simulated test traffic is sent to the large-scale firewall system at a preset traffic rate. Simultaneously, the generated traffic is format-adapted to ensure that the simulated test traffic maintains consistency with the protocol, format, and data structure of real business traffic, and conforms to the traffic inspection specifications of the large-scale firewall system. Simulated test traffic is sent strictly according to the parameters configured in the test cases. Automatic sending of single-round and multi-round simulated test traffic is supported. Traffic sending operations can be executed in an orderly manner according to test case priority and test round requirements. The traffic sending status is monitored in real time during the sending process. For situations such as traffic sending interruption or abnormality, a retry mechanism is triggered and error information is fed back to ensure the continuity and stability of traffic sending. If all retries fail, the test is stopped, and the process is preserved.

[0057] Optionally, while sending simulated test traffic, the system can monitor each traffic detection result returned by the large-scale firewall system in real time, and collect, organize, and store the traffic detection results in a preset database. The traffic detection results should include at least whether the traffic was blocked, the reason for blocking, the reason for allowing traffic, and the detection time.

[0058] Specifically, while sending simulated test traffic, the system monitors the detection results of every traffic from the large-scale firewall system in real time, including but not limited to whether the traffic was blocked, the reason for blocking (blocked by policy), the reason for allowing it (such as whitelist), and the detection time. At the same time, the system collects, organizes, and stores the results data in a database (e.g., InfluxDB) to ensure that the inspection results are not lost, thereby achieving real-time traffic detection result reception and collection.

[0059] In some implementations, key metrics throughout the simulated test traffic transmission process are monitored in real time; traffic parameters are dynamically adjusted based on these key metrics, and the simulated test traffic is then sent to the large-scale firewall system based on the adjusted traffic parameters. These key metrics include at least the transmission rate, the number of transmissions, and the percentage of successful transmissions.

[0060] Specifically, it can monitor the entire traffic transmission process in real time, including key indicators such as transmission rate, number of transmissions, and percentage of successful transmissions. It also supports dynamic adjustment of traffic transmission parameters based on test execution (such as temporarily adjusting the transmission frequency or pausing / resuming traffic transmission) to adapt to the dynamic changes in test scenarios.

[0061] Therefore, by distributing the pre-configuration of the large-scale firewall system through configuration parameters, including enabling rules and policies, setting interception thresholds, and entering whitelists, the large-scale firewall system is ensured to be in the initial state required for testing, achieving accurate matching of protection policies in the test scenario; during the distribution process, the distribution progress and status are fed back in real time, and an error prompt and retry mechanism are provided for parameters that fail to be distributed, ensuring that all configurations required for testing can be successfully distributed to the large-scale firewall system.

[0062] S105 collects target test data throughout the entire testing process, performs multi-dimensional processing on the target test data, calculates the corresponding core evaluation indicators based on the processed target test data, and generates the corresponding visual test report based on the core evaluation indicators.

[0063] In this embodiment, the target data includes at least test case execution data, traffic transmission data, large-scale firewall system detection and interception data, and traffic detection results; multi-dimensional processing includes at least deduplication, verification, and integration; core evaluation metrics include at least test case coverage, interception accuracy, false negative / false positive data, and test execution efficiency. The target test data throughout the entire testing process undergoes multi-dimensional processing, and core evaluation metrics are calculated. A visual test report is generated based on these core evaluation metrics. For example, such as... Figure 2 As shown.

[0064] Specifically, the system collects relevant test data throughout the entire testing process, including but not limited to test case execution data, traffic transmission data, and large-scale firewall system detection and interception data. It then performs data deduplication, verification, and integration to ensure the integrity of the aggregated data. Based on the aggregated test data, it automatically calculates various core evaluation metrics, including test case coverage (such as the coverage percentage of core scenario test cases and the coverage percentage of all scenario test cases), interception accuracy, false negative / false positive data (such as the number of false negatives, the false negative rate, the number of false positives, and the false positive rate), and test execution efficiency (such as the time spent on a single round of testing and the time spent on full-scale testing). This provides a clear and intuitive presentation of the large-scale firewall system's test performance, facilitating report interpretation.

[0065] In addition, this application supports generating visual test reports in various formats (such as PDF and HTML), which can use a combination of charts (bar charts, line charts, pie charts, etc.) and text to intuitively display test indicators, test conclusions, and anomalies in various dimensions, facilitating the quick acquisition of key information. Furthermore, it supports viewing historical test reports, making it convenient for reviewing and comparing results from multiple tests.

[0066] The text interaction traffic robustness automated testing method provided in this application configures different scenarios based on the test target, and combines the configurations of different scenarios to obtain test cases covering different scenarios. For the test cases, multiple test case parameters are configured, and the test cases are grouped and prioritized based on test requirements. The configuration parameters required for the test are processed to obtain configuration parameters that meet the parameter receiving requirements of the large-scale firewall system, and the configuration parameters are sent to the large-scale firewall system. The large-scale firewall system is pre-configured based on the configuration parameters. Based on the test case parameters, simulated test traffic is sent to the large-scale firewall system for traffic detection, and the traffic detection results returned by the large-scale firewall system are received. The target test data of the entire test process is collected, and the target test data is processed in multiple dimensions. Based on the processed target test data, the corresponding core evaluation indicators are calculated, and the corresponding visual test report is generated based on the core evaluation indicators. This application presents an automated testing method for the robustness of text interaction traffic. It generates test cases covering different scenarios and configures various parameters for these test cases. The method pre-configures a large-scale firewall system using these parameters and sends simulated test traffic to the firewall system according to the test case parameters for traffic detection. Finally, it calculates core evaluation metrics based on data from the entire testing process and generates a visual test report. This method comprehensively covers different scenarios, increases the number of words and characters covered, and enables comprehensive verification of massive attack scenarios, avoiding blind spots in firewall defenses. Furthermore, it eliminates reliance on manual intervention, improving testing effectiveness and efficiency while reducing testing and maintenance costs.

[0067] Furthermore, based on the test objective, a corresponding configuration range is selected; based on the configuration range, corresponding configuration parameters are selected as the configuration parameters required for the test and sent to the large-scale firewall system.

[0068] Specifically, this application can flexibly select the scope of pre-configuration according to the needs of this test, such as selecting all configurations, a single configuration (such as enabling only rule policies), or multiple configurations, thereby achieving precise configuration and batch configuration.

[0069] Therefore, this application forms a fully automated testing technology solution for the entire process of "test case setting - test case configuration - configuration distribution - traffic sending - report generation". It solves the technical defects of manual test case output, such as incompleteness, large error, low efficiency, single traffic simulation, and lack of test result archiving. It significantly improves the accuracy, efficiency and standardization of testing, and provides more reliable and efficient technical support for the performance verification and optimization upgrade of large-scale firewall systems. It has significant technical advantages and practical value.

[0070] Figure 3This is a schematic diagram of the structure of an automated text interaction traffic robustness testing device provided according to an embodiment of this application; as shown below. Figure 3 As shown, the automated text interaction traffic robustness testing device 300 of this application embodiment may specifically include: The acquisition module 301 is used to configure different scenarios based on the test target, and to combine the configurations of different scenarios to obtain test cases covering different scenarios; Configuration module 302 is used to configure various test case parameters for test cases, group test cases based on test requirements, and prioritize test cases. The distribution module 303 is used to process the configuration parameters required for testing to obtain configuration parameters that meet the parameter receiving requirements of the large-scale firewall system, and distribute the configuration parameters to the large-scale firewall system to pre-configure the large-scale firewall system based on the configuration parameters. The detection module 304 is used to send simulated test traffic to the large-scale firewall system for traffic detection based on the test case parameters, and to receive the traffic detection results returned by the large-scale firewall system; wherein, the simulated test traffic represents the simulation of real business traffic; The generation module 305 is used to collect target test data throughout the entire testing process, perform multi-dimensional processing on the target test data, calculate the corresponding core evaluation indicators based on the processed target test data, and generate the corresponding visual test report based on the core evaluation indicators.

[0071] In one possible implementation, the acquisition module is specifically used for: The system randomly generates multiple words covering different scenarios from a preset word library, scrambles the words, and then splices them together; it also randomly extracts multiple Chinese characters from a preset Chinese character library, scrambles the characters, and then splices them together. Configure punctuation marks from a preset punctuation mark library, configure Chinese special characters from a preset special character library, and randomly generate multiple words from a preset word library.

[0072] In one possible implementation, the acquisition module is specifically used for: The concatenated words, Chinese characters, punctuation marks, and special Chinese characters are randomly combined to generate multiple original elements. These original elements are then shuffled and concatenated to obtain the corresponding first and second test cases. The concatenated words, Chinese characters, punctuation marks, special Chinese characters and words are randomly combined to generate multiple original elements. The original elements are then shuffled and concatenated to obtain the corresponding third test case. For any test case, multiple original elements are generated, the original elements are shuffled and then spliced ​​together, and the spliced ​​result is randomly encoded for testing to obtain the corresponding fourth test case; the fourth test case represents the escape scenario under the protection capability.

[0073] In one possible implementation, the apparatus further includes: The verification module is used to verify the actual configuration of the large-scale firewall system based on the configuration parameters after the configuration parameters are sent to the large-scale firewall system, and to compare the consistency between the configuration parameters and the actual configuration parameters of the large-scale firewall system. The resend module is used to automatically resend configuration parameters when the configured parameters and the actual configuration parameters are inconsistent.

[0074] In one possible implementation, the detection module is specifically used for: Adjust the format of the generated simulated test traffic to match the real business traffic and adapt it to the traffic inspection standards of the large-scale firewall system. Based on the test case parameters configured in the test case, the corresponding traffic parameters are determined. Simulated test traffic is sent to the large-scale firewall system based on the traffic parameters, and the traffic sending status is monitored in real time during the sending process. If the traffic transmission status is abnormal, the simulated test traffic will be resent and the abnormal information will be reported back.

[0075] In one possible implementation, the apparatus further includes: The monitoring module is used to monitor key indicators of the entire process of simulated test traffic transmission in real time; among which, key indicators include at least transmission rate, number of transmissions, and percentage of successful transmissions. The adjustment module is used to dynamically adjust traffic parameters based on key indicators, and send simulated test traffic to the large-scale firewall system based on the adjusted traffic parameters.

[0076] In one possible implementation, the apparatus further includes: The first selection module is used to select the corresponding configuration range based on the test objective; The second selection module is used to select the corresponding configuration parameters based on the configuration range as the configuration parameters required for testing and send them to the large-scale firewall system.

[0077] The automated text interaction traffic robustness testing device provided in this application configures different scenarios based on the test objectives, combines the configurations of different scenarios to obtain test cases covering different scenarios, configures multiple test case parameters for each test case, groups test cases based on test requirements, and prioritizes test cases. It processes the configuration parameters required for testing to obtain configuration parameters that meet the parameter receiving requirements of a large-scale firewall system, and sends the configuration parameters to the large-scale firewall system. Based on the configuration parameters, the large-scale firewall system is pre-configured. Based on the test case parameters, simulated test traffic is sent to the large-scale firewall system for traffic detection, and the system receives the traffic detection results returned by the large-scale firewall system. It collects target test data throughout the entire testing process, performs multi-dimensional processing on the target test data, calculates corresponding core evaluation indicators based on the processed target test data, and generates corresponding visual test reports based on the core evaluation indicators. This application presents an automated testing device for robustness of text interaction traffic. It generates test cases covering different scenarios and configures various parameters for these test cases. By pre-configuring the large-scale firewall system using these parameters, and sending simulated test traffic to the large-scale firewall system according to the test case parameters for traffic detection, it calculates core evaluation indicators based on the data from the entire testing process and generates a visual test report. This device comprehensively covers different scenarios, increases the number of words and characters covered, and enables comprehensive verification of massive attack scenarios, avoiding blind spots in firewall defense. Furthermore, it eliminates reliance on manual intervention, improving testing effectiveness and efficiency while reducing testing and maintenance costs.

[0078] like Figure 4 As shown in the embodiment of this application, an electronic device 400 includes a processor 401, a memory 402, and a bus. The memory 402 stores machine-readable instructions executable by the processor 401. When the electronic device is running, the processor 401 communicates with the memory 402 via the bus. The processor 401 executes the machine-readable instructions to perform the steps of the text interaction traffic robustness automated testing method described above.

[0079] Specifically, the memory 402 and processor 401 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 401 runs the computer program stored in the memory 402, it can execute the above-mentioned automated test method for robustness of text interaction traffic.

[0080] Corresponding to the above-described automated testing method for text interaction traffic robustness, this application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the above-described automated testing method for text interaction traffic robustness.

[0081] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0082] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0084] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the deployment methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0085] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An automated testing method for the robustness of text interaction traffic, characterized in that, The method includes: Based on the test objectives, different scenarios are configured, and the configurations of different scenarios are combined to obtain test cases that cover different scenarios; For the test cases, configure various test case parameters, group the test cases based on test requirements, and prioritize the test cases. The configuration parameters required for testing are processed to obtain configuration parameters that meet the parameter receiving requirements of the large-scale firewall system, and the configuration parameters are sent to the large-scale firewall system. The large-scale firewall system is then pre-configured based on the configuration parameters. Based on the test case parameters, simulated test traffic is sent to the large-scale firewall system for traffic detection, and the traffic detection results returned by the large-scale firewall system are received; wherein, the simulated test traffic represents the simulation of real business traffic; Collect target test data throughout the entire testing process, perform multi-dimensional processing on the target test data, calculate the corresponding core evaluation indicators based on the processed target test data, and generate the corresponding visual test report based on the core evaluation indicators.

2. The method according to claim 1, characterized in that, The different configuration scenarios include: The system randomly generates multiple words covering different scenarios from a preset word library, scrambles the words, and then splices them together; it also randomly extracts multiple Chinese characters from a preset Chinese character library, scrambles the characters, and then splices them together. Configure punctuation marks from a preset punctuation mark library, configure Chinese special characters from a preset special character library, and randomly generate multiple words from a preset word library.

3. The method according to claim 2, characterized in that, The combination of configurations for different scenarios to obtain test cases covering different scenarios includes: The concatenated words, Chinese characters, punctuation marks, and special Chinese characters are randomly combined to generate multiple original elements. The original elements are then shuffled and concatenated to obtain the corresponding first test case and second test case. The concatenated words, Chinese characters, punctuation marks, Chinese special characters, and words are randomly combined to generate multiple original elements. These original elements are then shuffled and concatenated to obtain the corresponding third test case. For any test case, multiple original elements are generated, the original elements are shuffled and then spliced ​​together, and the splicing result is randomly encoded for testing to obtain the corresponding fourth test case; wherein, the fourth test case represents the escape scenario under the protection capability.

4. The method according to claim 1, characterized in that, After the configuration parameters are sent to the large-scale firewall system, the method further includes: The actual configuration of the large-scale firewall system is verified based on the configuration parameters, and the consistency between the configuration parameters and the actual configuration parameters of the large-scale firewall system is compared. If the configuration parameters and the actual configuration parameters are inconsistent, the configuration parameters will be automatically resent.

5. The method according to claim 1, characterized in that, Sending the simulated test traffic to the large-scale firewall system includes: Adjust the format of the generated simulated test traffic to match that of real business traffic and adapt it to the traffic inspection standards of the large model firewall system. Based on the test case parameters configured in the test case, the corresponding traffic parameters are determined, and the simulated test traffic is sent to the large model firewall system based on the traffic parameters, while the traffic sending status is monitored in real time during the sending process. If the traffic transmission status is abnormal, the simulated test traffic will be retransmitted and the abnormal information will be fed back.

6. The method according to claim 5, characterized in that, The method further includes: Real-time monitoring of key indicators throughout the entire process of simulated test traffic transmission; wherein, the key indicators include at least transmission rate, number of transmissions, and percentage of successful transmissions; The traffic parameters are dynamically adjusted based on the key indicators, and the simulated test traffic is sent to the large-scale firewall system based on the adjusted traffic parameters.

7. The method according to claim 1, characterized in that, The method further includes: Based on the aforementioned test objectives, select the corresponding configuration range; Based on the configuration range, the corresponding configuration parameters are selected as the configuration parameters required for testing and sent to the large-scale firewall system.

8. An automated testing device for the robustness of text interaction traffic, characterized in that, The device includes: The acquisition module is used to configure different scenarios based on the test objectives, and to combine the configurations of different scenarios to obtain test cases covering different scenarios; The configuration module is used to configure various test case parameters for the test cases, group the test cases based on test requirements, and prioritize the test cases. The distribution module is used to process the configuration parameters required for testing to obtain configuration parameters that meet the parameter receiving requirements of the large-scale firewall system, and distribute the configuration parameters to the large-scale firewall system to pre-configure the large-scale firewall system based on the configuration parameters. The detection module is used to send simulated test traffic to the large-scale firewall system for traffic detection based on the test case parameters, and to receive the traffic detection results returned by the large-scale firewall system; wherein, the simulated test traffic represents the simulation of real business traffic; The generation module is used to collect target test data throughout the entire testing process, perform multi-dimensional processing on the target test data, calculate the corresponding core evaluation indicators based on the processed target test data, and generate the corresponding visual test report based on the core evaluation indicators.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the automated text interaction traffic robustness testing method as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the automated text interaction traffic robustness testing method as described in any one of claims 1 to 7.

Images