Set top box automatic testing method and system based on big data

Abstract

The invention discloses a set top box automatic testing method and system based on big data, and mainly relates to the technical field of set top box automatic testing. Comprising the steps that log collection is conducted through a built-in lightweight agent of an equipment system, and log data information is extracted; performing data cleaning on the extracted data information through a filtering rule to form a data set; performing K-means clustering on the data set subjected to data cleaning to extract a behavior pattern, and constructing a defect prediction model; the extracted log data information is divided into a training set and a verification set, and a defect prediction model is used for training; and generating a core test case and a boundary test case. The method has the beneficial effects that the test strategy can be dynamically optimized while the test case better fits the real use scene of the user.

Description

Technical Field

[0001] The present invention relates to the technical field of set-top box automated testing, and in particular to a set-top box automated testing method and system based on big data. Background Art

[0002] Currently, the set-top box automation test mainly writes or records test cases through preset test scenarios, covering fixed operation paths based on predefined remote control button commands (such as "right button → down button → confirm") and UI screenshot comparison.

[0003] Using the above method on multiple devices can indeed improve test efficiency, but there are also the following problems: 1. Insufficient test case coverage: Existing solutions rely on predefined test cases, which are usually based on standard scenarios or developers' assumptions and cannot fully reflect the usage habits of real users. For example, users may frequently use the live broadcast function during specific time periods (such as weekend nights), but the test cases do not cover such scenarios. 2. Static test strategy: Once the test cases and strategies are recorded, subsequent adjustments are difficult and cannot be dynamically optimized according to changes in user behavior. This will cause the test results to gradually become out of touch with the actual usage scenarios, especially in the context of rapid changes in user habits. 3. Low data utilization: Although the test platform can collect set-top box operation data (such as logs and error reports), this data is only used to generate static reports and is not deeply analyzed or applied to the optimization of the test process, wasting potential value. 4. Mismatch between test efficiency and accuracy: Existing technologies usually adopt a unified test strategy when testing a large number of devices, ignoring individual differences (such as network conditions and device model differences among users in different regions), resulting in inaccurate test results and low efficiency in some scenarios.

[0004] The main reasons for these problems are: 1. Lack of dynamic feedback mechanism: The existing system does not effectively link the actual user usage data with the testing process, resulting in the test case design being too static and unable to adapt to diverse user behaviors; 2. Insufficient data analysis capabilities: Although the existing test platform has data collection capabilities, it lacks big data processing and analysis capabilities and is unable to extract meaningful data from massive data; 3. Low technical integration: Automated testing and big data analysis technologies have not been deeply integrated in existing solutions, which limits the intelligence and adaptability of the testing process.

[0005] Therefore, there is an urgent need for an automated set-top box testing method and system based on big data to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to provide a set-top box automatic testing method and system based on big data, which can realize dynamic optimization of testing strategies while making test cases more in line with users' actual usage scenarios.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions: On the one hand, the present invention provides a set-top box automated testing method based on big data, comprising the following steps: S1: Log data collection is performed through the lightweight agent built into the device system; S2: Clean the extracted data information through filtering rules to form a data set; S3: Use K-means clustering to extract behavior patterns from the cleaned data set, and use lightweight random forest to build a defect prediction model; S4: Divide the log data information extracted in step S1 into a training set and a validation set, and train them using the defect prediction model; S5: Generate core test cases and boundary test cases.

[0008] Preferably, the step S1 comprises: S11: extract key information of the log, the key information includes: device ID, time point, and operation behavior; S12: Set the data collection interval and the number of abnormal retries; S13: Alternately execute instructions for data collection and data transmission.

[0009] Preferably, the step S2 comprises: S21: Remove records with missing or incorrect timestamps, delete duplicate data and invalid operation paths, and if two records are found to have the same time, keep the latest one; S22: Setting a coding mapping table of the operation path; S23: extracting time series features, calculating the average operation intervals of the last ten times and the cumulative usage time of the day; S24: Normalize the values ​​of duration, network delay, etc. ; S25: Evaluate the health of the device through log data: ; in, To evaluate the health of the equipment, A warning reminder is triggered.

[0010] Preferably, extracting behavior patterns from the data set after data cleaning using K-means clustering in step S3 includes: Initialize K cluster centers and calculate the Euclidean distance from each data point to the center: the distance formula is: ; Assign data points to the nearest center, update the center position to the cluster mean, and iterate until the center no longer changes or the maximum number of iterations is reached; Output 5 types of behavior patterns; The optimal K value is determined by the elbow rule, based on the WSS curve.

[0011] Preferably, the step S4 comprises: S41: Divide the log data information extracted in step S1 into a training set and a validation set in a ratio of 8 to 2 for training, and determine the optimal tree depth and number through grid search; S42: Add daily test result data to the training process, and perform full retraining when the accuracy drops by more than 2%; S43: Perform time series anomaly detection and calculate the daily average network delay and standard deviation , calculate for each record value: ,like , it is marked as abnormal; S44: Construction priority algorithm: ; S45: Build a test task generation rule base and execute it from high to low priority.

[0012] Preferably, the step S5 is specifically as follows: based on the behavior pattern, mapping table, and priority sorting, core test cases and boundary test cases are generated respectively, wherein the core test cases are based on high-frequency behavior patterns, and the boundary test cases are based on abnormal patterns.

[0013] Preferably, the method further includes executing a detection process on the test case, including: Check the syntax and executability of test cases through the API interface. If failed, log and retry the generation. Receive successfully generated test cases and assign combinations of test cases and devices based on parameters such as device model, performance, and preset rules; Concurrent testing of each proxy node Devices, split the received test cases into test steps, identify the device model corresponding to the test case, and distribute them in a designated manner; During the test execution, the agent node continuously feeds back the device's operation log and abnormal conditions to the automated testing platform, and automatically generates a visual report and result analysis after the test is completed.

[0014] On the other hand, a test system based on the above-mentioned set-top box automated test method based on big data is provided, comprising: The data collection module is used to collect log data through the lightweight agent built into the device system; The data processing module is used to: clean the extracted data information through filtering rules to form a data set; Model building module, used to: extract behavior patterns from the cleaned data set using K-means clustering, and build a defect prediction model using lightweight random forest; The model training module is used to: divide the extracted log data information into a training set and a validation set, and use the defect prediction model for training; Data output module, used to generate core test cases and boundary test cases.

[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. Test cases are more in line with users' real usage scenarios: The present invention uses big data to analyze the operation logs of equipment to obtain high-frequency and special operation behaviors, and combines the automated testing platform to generate test cases with different priorities, thereby improving the coverage of test scenarios and the accuracy of tests.

[0016] 2. Dynamic optimization of test strategies: The loop mechanism of the feedback optimization unit of the present invention enables the test strategies to be dynamically adjusted continuously according to the test results, making them more accurate and reasonable.

[0017] 3. Full utilization of log data: The present invention combines data collection with big data analysis to make the operation log of the equipment more valuable, rather than just being used for report presentation of conventional automated testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a flowchart of data preprocessing of the present invention; DETAILED DESCRIPTION The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0019] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.

[0020] Example: like Figure 1 As shown, this embodiment provides a set-top box automated testing method based on big data, comprising the following steps: S1: Log data collection is performed through the lightweight agent built into the device system; S2: Clean the extracted data information through filtering rules to form a data set; S3: Use K-means clustering to extract behavior patterns from the cleaned data set, and use lightweight random forest to build a defect prediction model; S4: Divide the log data information extracted in step S1 into a training set and a validation set, and train them using the defect prediction model; S5: Generate core test cases and boundary test cases.

[0021] In step S1, log data is collected through the lightweight agent built into the device system, including: S11: Extract key information of the log, which includes: device ID, time point, operation behavior, etc. It can also include metadata such as device model, firmware version, geographic location, etc. to support more fine-grained analysis, package the log into a preset format, encrypt sensitive information using AES-256, and use SHA-256 verification to prevent data tampering. In this embodiment, the log example is as follows: { "device_id": "STB001", "timestamp": "2025-02-10 10:00:00", "channel_switch": 5, "app_usage": {"vod": 1200, "live": 300}, "network_delay": 50, "errors": ["buffer_timeout"] }; S12: Set the data collection interval and the number of abnormal retries: The data collection interval is set to 5 minutes, and the abnormal retries are set to 3 times. After the collection is completed, ensure that the data packet MB, during peak hours (e.g., at night), the amount of data may increase, and the agent automatically adjusts the frequency to 3 minutes to avoid network congestion caused by large data packets; S13: Alternately execute instructions for data collection and data transmission, specifically: within the interval of the task of data collection, send the data to the big data analysis platform, connect to the platform for 10s, retry for 5 minutes, retry up to 3 times, cache locally after failure. The maximum cache is 10MB, and resend after the network is restored.

[0022] Step S2 is specifically as follows: The big data platform cleans the massive data through filtering rules, removes records with missing or incorrect timestamps, deletes duplicate data (based on device_id and timestamp) and invalid operation paths, and retains the latest one if two records are found to have the same time. Set the operation path coding mapping table, such as "power on" = 1, "on-demand" = 2, "movie" = 3, etc.; Extract time series features, calculate the average operation interval of the last 10 times and the cumulative usage time of the day, etc.; Normalize the values ​​of duration, network delay, etc. to [0,1]: Formula: : The network delay range is [10, 200], 50 is normalized to (50-10) / (200-10)=0.21, and derived features are extracted, such as the total daily usage time and average network delay; The health of the equipment is evaluated through data. The calculation rules are as follows:

[0023] When HealthScore < 0.6, a warning reminder is triggered.

[0024] Step S3 is specifically: K-means clustering is used to extract behavior patterns from the cleaned data set. Features include path, switching frequency, duration, network delay, etc. The main process is as follows: 1) Initialize K cluster centers (K=5, randomly selected) and calculate the Euclidean distance from each data point to the center: Distance formula: ; 2) Assign the data points to the nearest center, update the center position to the cluster mean, and iterate until the center no longer changes or the maximum number of iterations (100 times) is reached; 3) Output behavior patterns, such as: Cluster 1: nighttime live broadcast users (high live broadcast duration), Cluster 2: daytime on-demand users (high on-demand duration), high-frequency switching users, low-latency users, peak-time users, users with poor performance, etc., support dynamic adjustment of the K value, and determine the optimal number of clusters through the Elbow Method; 4) Based on the within-cluster sum of squares (WSS) curve, the optimal K value is determined by the elbow method (ElbowMethod); Build a defect prediction model using lightweight random forest, number of decision trees = 50, maximum depth = 5. Feature vector: feature_vector = [ path_encoded, # Encoded operation path (such as [1,2,3]) health_score,# Health score (0.72) network_type,# Network type (0=5G, 1=broadband) time_since_last_reboot# Time since last reboot (8 hours) ].

[0025] Step S4 is specifically: The log data is trained with a ratio of 80% for training set and 20% for validation set, and grid search is used to determine the optimal tree depth and number. Output example: {"device_id": "STB-001", "P_defect": 0.85, "risk_type": "memory overflow"}; Add daily test result data to the training set, and retrain the entire set when the accuracy drops by more than 2%; Perform time series anomaly detection (Z-score) and calculate the daily average network latency and standard deviation , calculate the Z value for each record: ,like , marked as abnormal (such as sudden increase in network delay), example: μ=50ms, σ=10ms, x=85ms, Z=(85-50) / 10=3.5, marked as abnormal; Construct a priority algorithm, the calculation formula is: ; Indicates the probability of defect occurrence, indicating the possibility of defects in the current test target. Input feature vectors (operation path, health, network type, etc.) into the random forest model and output probability values ​​(0~1), such as: Indicates an 85% probability of triggering a defect; Indicates the operation frequency, which is the number of times a test target is triggered within a statistical period, such as: the path "channel switching" is triggered 50 times within 10 minutes; Indicates the maximum frequency, that is, the highest triggering number among all test targets in the same statistical period. Sort the frequencies of all operation paths or devices in the current period and take the maximum value. 0.7 and 0.3 are weight coefficients, which are used to balance defect risk and operation frequency. That is, if OperationFrequency of a path is 150 times / hour (MaxFrequency is 200), P_defect is 0.85, then Priority is 0.7×0.85+0.3×(150 / 200)=0.805. Build a test task generation rule base and execute it in descending order of priority: { "task_type": "Stress Test", "trigger_condition": "HealthScore<0.6&&P_defect>0.8", "parameters": { "concurrency": 10, "resolution": "3840x2160", "duration": "300s" } }.

[0026] Step 5, specifically: Based on the calculated behavior patterns + rule mapping + priority sorting, core test cases and boundary test cases are generated respectively. Core test cases are based on high-frequency behavior patterns (such as frequent channel switching); boundary test cases are based on abnormal patterns (such as network interruption). The process is as follows: 1) Mapping behavior patterns to test targets, such as: "nighttime live broadcast users" → test live broadcast fluency, "abnormal network delay" → test network disconnection recovery; 2) Calculate the priority: Priority = usage frequency × abnormal weight (the default abnormal weight is 1, and the abnormal scenario is 2); Example: Live broadcast usage frequency is 0.8, exception weight is 1, and priority = 0.8.

[0027] In this embodiment, the API interface can also be used to check the syntax and executability of the test case, support JavaScript and Python, etc. If it fails, log it and retry the generation; The automated testing platform receives the successfully generated test cases and assigns and combines the test cases with the equipment according to parameters such as equipment model, performance, and preset rules; The test cases are distributed to the agent nodes corresponding to the set-top boxes, waiting for the agent nodes to execute the tests; Each proxy node tests <10 devices concurrently, splits the received test cases into test steps, identifies the device model corresponding to the test case, and distributes them in a designated manner; During the test execution, the agent node continuously feeds back the device's operation logs and abnormal conditions to the automated test platform, and automatically generates a visual report and result analysis after the test is completed; Analyze test results regularly, calculate failure rate = number of failed use cases / total number of use cases, and identify scenarios with high failure rates (such as the percentage of use cases with response time > 1 second); Combined with the analysis results of the previous step, adjust the weight to increase the clustering weight of the failure scenario. The new weight = original weight × (1 + failure rate). Example: original weight 1, failure rate 0.2, new weight = 1.2; Rerun K-means to generate new use cases. You can also use reinforcement learning to adjust the weights and predict future optimization directions based on historical failure rates. The testing strategy can be optimized based on user feedback. For example, if customer service records show that users report that live broadcasts are stuck at night, the weight of the night live broadcast test can be increased accordingly.

[0028] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A set-top box automated testing method based on big data, characterized in that: The following steps are involved: S1: Log data collection is performed through the lightweight agent built into the device system; S2: Clean the extracted data information through filtering rules to form a data set; S3: Use K-means clustering to extract behavior patterns from the cleaned data set, and use lightweight random forest to build a defect prediction model; S4: Divide the log data information extracted in step S1 into a training set and a validation set, and train them using the defect prediction model; S5: Generate core test cases and boundary test cases.

2. The method for automated testing of set-top boxes based on big data according to claim 1, characterized in that: The step S1 comprises: S11: extract key information of the log, the key information includes: device ID, time point, and operation behavior; S12: Set the data collection interval and the number of abnormal retries; S13: Alternately execute instructions for data collection and data transmission.

3. The set-top box automated testing method based on big data according to claim 1, characterized in that: The step S2 comprises: S21: Remove records with missing or incorrect timestamps, delete duplicate data and invalid operation paths, and if two records are found to have the same time, keep the latest one; S22: Setting a coding mapping table of the operation path; S23: extracting time series features, calculating the average operation intervals of the last ten times and the cumulative usage time of the day; S24: Normalize duration and network delay values ​​to ; S25: Evaluate the health of the device through log data: ； in, To evaluate the health of the equipment, A warning reminder is triggered.

4. The set-top box automated testing method based on big data according to claim 1, characterized in that: The step S3 of extracting behavior patterns from the data set after data cleaning using K-means clustering includes: Initialize K cluster centers and calculate the Euclidean distance from each data point to the center: the distance formula is: ; Assign data points to the nearest center, update the center position to the cluster mean, and iterate until the center no longer changes or the maximum number of iterations is reached; Outputting a behavior pattern according to the result of clustering the data set, the behavior pattern includes but is not limited to: nighttime live broadcast users, daytime on-demand users, high-frequency switching users, low-latency users, peak-time users, and users with poor performance; Based on the WSS curve, the optimal K value is determined by the elbow rule.

5. The set-top box automated testing method based on big data according to claim 1, characterized in that: The step S4 comprises: S41: Divide the log data information extracted in step S1 into a training set and a validation set in a ratio of 8 to 2 for training, and determine the optimal tree depth and number through grid search; S42: Add daily test result data to the training set, and perform full retraining when the accuracy drops by more than 2%; S43: Perform time series anomaly detection and calculate the daily average network delay and standard deviation , calculate for each record value: ,like , it is marked as abnormal; S44: Construction priority algorithm: ； in, Indicates the probability of defect occurrence, indicating the possibility of defects in the current test target; Indicates the operation frequency, which is the number of times a test target is triggered within a statistical period; Indicates the maximum frequency, that is, the highest number of triggers among all test targets in the same statistical period. 0.7 and 0.3 are weight coefficients used to balance defect risk and operation frequency. S45: Build a test task generation rule base and execute it from high to low priority.

6. The set-top box automated testing method based on big data according to claim 1, characterized in that: The step S5 is specifically as follows: based on the behavior pattern, the mapping table, and the priority sorting, a core test case and a boundary test case are generated respectively, wherein the core test case is based on the high-frequency behavior pattern, and the boundary test case is based on the abnormal pattern.

7. The method for automated testing of set-top boxes based on big data according to claim 1, characterized in that: It also includes the execution of the detection process for the test cases, including: Check the syntax and executability of test cases through the API interface. If failed, log and retry the generation. Receive successfully generated test cases and assign combinations of test cases and devices based on device models, performance, and preset rule parameters; Concurrent testing of each proxy node Devices, split the received test cases into test steps, identify the device model corresponding to the test case, and distribute them in a designated manner; During the test execution, the agent node continuously feeds back the device's operation log and abnormal conditions to the automated testing platform, and automatically generates a visual report and result analysis after the test is completed.

8. A test system based on the set-top box automated test method based on big data as claimed in claim 1, characterized in that: include: The data collection module is used to collect log data through the lightweight agent built into the device system; The data processing module is used to: clean the extracted data information through filtering rules to form a data set; Model building module, used to: extract behavior patterns from the cleaned data set using K-means clustering, and build a defect prediction model using lightweight random forest; The model training module is used to: divide the extracted log data information into a training set and a validation set, and use the defect prediction model for training; Data output module, used to generate core test cases and boundary test cases.

Images