Automatic test monitoring method and system

Abstract

The present invention relates to the field of random numbers in environmental reliability tests, and in particular relates to an automatic test monitoring method and system. The test monitoring method comprises the steps of: implementing service logic of a reliability test by means of an automation tool; launching a plurality of background tasks, and reading and recording test metric data at every preset time interval; publishing the metric data to a specific topic by means of an MQTT protocol; processing and analyzing the test metric data; and displaying the processed test metric data in a visualized manner. The present invention not only improves the degree of automation and monitoring efficiency of an environmental reliability test of hardware, but also significantly enhances the sensitivity and response speed for various anomaly behaviors during the test, and provides strong technical support for improving the overall quality and reliability of electronic products.

Description

Automated test monitoring method and system TECHNICAL FIELD

[0001] The present application relates to the field of environmental reliability testing random numbers, in particular to an automated test monitoring method and system. BACKGROUND

[0002] With the rapid rise of the complexity of electronic product hardware and software systems, the standards for their environmental reliability have also been tightened. Traditional manual environmental reliability testing methods have been difficult to meet current needs due to low efficiency and the risk of missing critical issues. These problems are not only limited to immediate omissions during testing, but also due to insufficient monitoring, making it difficult to detect problems in a timely manner, especially those that require time accumulation to manifest.

[0003] In view of the above challenges, the introduction of automated testing methods is particularly urgent and important, aiming to assist and optimize manual testing processes, and improve testing efficiency and problem detection rate. Currently, although the industry is still in the manual testing stage, mainly relying on index abnormalities after long-term execution as the basis for result evaluation, this approach is obviously difficult to capture subtle abnormalities and potential long-term cumulative problems during testing.

[0004] SUMMARY

[0005] The present application provides an automated test monitoring method and system to solve the technical problems of low testing efficiency and poor real-time data of traditional environmental reliability testing methods.

[0006] The technical solution of the present application to solve the above technical problems is as follows:

[0007] An automated test monitoring method is provided, the steps of the test monitoring method comprising:

[0008] Implementing the business logic of reliability testing through an automated tool;

[0009] Starting multiple background tasks, reading and recording test index data at intervals of a preset time;

[0010] Publishing the index data to a specific topic through the MQTT protocol;

[0011] Processing and analyzing the test index data;

[0012] Visually displaying the processed test index data.

[0013] Further, the step of implementing the business logic of reliability testing through an automated tool comprises:

[0014] Initializing the test environment, including setting the parameters of the temperature and humidity chamber;

[0015] Configure the flow test equipment;

[0016] Start the device under test and initialize the configuration.

[0017] Further, the step of starting multiple background tasks and reading and recording test index data at intervals of a preset time comprises:

[0018] Read the temperature and humidity data;

[0019] Read the power output, feed voltage and USB voltage;

[0020] Read the wireless flow throughput and wired flow packet loss rate;

[0021] Read the channel temperature and POE load power;

[0022] Read the remote wireless network card rate.

[0023] Further, the step of publishing the index data to a specific topic through the MQTT protocol comprises:

[0024] Format the read index data into unified JSON data;

[0025] Design specific MQTT topics for different types of index data;

[0026] Automate tool connection to MQTT Broker agent;

[0027] Use the publish method of MQTT to publish the JSON data to the corresponding topic;

[0028] According to the importance of the data, set appropriate quality of service levels.

[0029] Further, the step of processing and analyzing the test index data comprises:

[0030] Clean and preprocess the raw data;

[0031] Calculate performance indicators;

[0032] Perform anomaly detection and alarm.

[0033] Further, the formula for calculating the performance indicators is:

[0034] where T is the current temperature, Tmax is the maximum allowed temperature; P is the current power, Pmax is the maximum allowed power; L is the current packet loss rate, Lmax is the maximum allowed packet loss rate; S is the current throughput, Smax is the maximum throughput; w1, w2, w3, w4 are weight coefficients.

[0035] In another aspect, there is provided an automated test monitoring system for implementing the automated test monitoring method as described above, the test monitoring system comprising:

[0036] an automated tool for implementing the business logic of the reliability test;

[0037] an MQTT Broker agent for managing the publishing and subscribing of MQTT messages;

[0038] a web backend server for processing and analyzing test data;

[0039] a web frontend server for providing user interface and data visualization; and

[0040] a database management server for storing and managing test data;

[0041] the automated tool comprises:

[0042] a temperature and humidity control module for controlling the temperature and humidity chamber;

[0043] a flow test module for controlling the flow test equipment;

[0044] a device configuration module for configuring the device under test.

[0045] Further, the test monitoring system further comprises:

[0046] a wireless network card control Windows service for reading the uplink and downlink rates of a remote wireless network card;

[0047] an upgrade server for version management and upgrade of the automated tool;

[0048] a high and low temperature chamber API server for providing a software interface for operating the temperature chamber;

[0049] a file management server for managing files generated during the test process;

[0050] a time calculation service for processing time-related data;

[0051] an SMS server for sending alarm information;

[0052] a log server for recording system logs;

[0053] a monitoring large screen for displaying the test status in real time; and

[0054] an internal network for connecting various components of the system.

[0055] Further, the web backend server comprises:

[0056] a data processing module for data cleaning and preprocessing;

[0057] an indicator calculation module for calculating key performance indicators; and

[0058] an anomaly detection module for detecting anomalies and triggering alarms.

[0059] Further, the Web front-end server comprises:

[0060] a data visualization module for generating charts and dashboards;

[0061] a user interaction module for processing user inputs and operations;

[0062] a real-time update module for updating display data in real time.

[0063] The present application has the following beneficial effects:

[0064] The present application not only improves the automation degree and monitoring efficiency of hardware environment reliability testing, but also significantly enhances the sensitivity and response speed to various abnormal behaviors in the testing process, thereby providing strong technical support for improving the overall quality and reliability of electronic products.

[0065] The above description is only a summary of the technical solutions of the present application. In order to make the technical means of the present application clearer and to enable the implementation according to the content of the description, the following describes the preferred embodiments of the present application in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0066] Fig. 1 is a flowchart of an automated test monitoring method according to an embodiment of the present application;

[0067] Fig. 2 is an architectural diagram of an automated test monitoring system according to an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. The examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application. In addition, it should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0069] The disclosure below provides many different embodiments or examples for implementing different structures of the application. For the purpose of simplifying the disclosure of the application, the components and settings of specific examples are described below. Of course, they are only examples and the purpose is not to limit the application. In addition, the application can repeatedly refer to numbers and / or letters in different examples, and such repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed. In addition, the application provides examples of various specific brands, models and series, but those of ordinary skill in the art can realize the use scenarios of other brands, models and series.

[0070] The application provides the following preferred embodiments:

[0071] Embodiment one

[0072] In order to solve the problem of low monitoring efficiency and poor real-time data of the existing automatic test system, the embodiment further optimizes the specific implementation steps of the automatic test monitoring method.

[0073] As shown in FIG. 1, the steps of the test monitoring method include:

[0074] S100, realizing the business logic of environmental reliability test through an automatic tool.

[0075] S200, starting multiple background tasks, reading and recording test index data at intervals of a preset time.

[0076] S300, publishing the index data to a specific topic through the MQTT protocol.

[0077] S400, processing and analyzing the test index data.

[0078] S500, visualizing the processed test index data.

[0079] Specifically, the automatic test monitoring method provided by the embodiment first realizes the business logic of reliability test through an automatic tool. The automatic tool can be developed by using programming languages such as Python and C++, has the ability to control hardware such as temperature and humidity environmental boxes and flow test equipment, and can execute various test tasks according to a preset test process.

[0080] Further, during the test execution, the automation tool initiates multiple background tasks that read and record various test metric data at preset time intervals (e.g., every 5 seconds, every 10 seconds, etc.). These metric data can include, but are not limited to: real-time temperature, humidity, alarm status of the temperature and humidity environmental chamber; output power of the power supply; feed voltage of the voice device; voltage of the USB device; throughput, packet loss rate of wired / wireless network; real-time temperature of each test point; power of POE device, etc. It should be understood that which specific metrics to monitor can be configured and selected by the test personnel according to actual needs.

[0081] Further, the automation tool publishes the collected metric data to a specific topic through the MQTT protocol. MQTT is a lightweight publish-subscribe message transmission protocol suitable for remote connection scenarios that require small code and limited bandwidth. It can be understood that transmitting data through the MQTT protocol has the advantages of strong real-time performance and low resource occupation, which is very suitable for the data transmission needs in this scenario.

[0082] Further, after the data is published, the data processing module in the system subscribes to the corresponding MQTT topic, receives and processes these test metric data. The processing process can include data cleaning, outlier detection, trend analysis, etc. For example, data curves can be smoothed by moving average method, abnormal fluctuations can be detected by setting threshold, or data change trend can be predicted by machine learning algorithm, etc.

[0083] Further, the processed test metric data is presented to the user in a visualized manner. Visualization can be achieved through web front-end technology, such as using ECharts or other chart libraries to draw real-time updated graphs, dashboards, etc. Users can understand the test progress in time and quickly find abnormal situations through these intuitive charts.

[0084] The benefits of this embodiment are that through the combination of automation tools, MQTT protocol and data visualization technology, real-time monitoring of the test process and immediate analysis of the data are achieved. This method not only improves the efficiency of test monitoring, but also enhances the ability to discover and respond to abnormal situations in time, thereby effectively improving the reliability and effectiveness of the entire automated test process. At the same time, the modular design of this method also provides convenience for the expansion and optimization of the system, and the monitoring indicators and analysis methods can be flexibly adjusted according to actual needs.

[0085] Embodiment Two

[0086] Further, the present application provides a tool and server system for automated testing to achieve the purpose of environmental reliability automated testing and monitoring.

[0087] I. Composition of the automated test monitoring system

[0088] The architecture of the automatic test monitoring system of the embodiment is shown in FIG. 2, which is composed of an automatic tool, a wireless network card control windows service, an upgrade server, a high and low temperature API server, a web background server, a file management server, a database management server, a time calculation service, a short message server, a log server, a monitoring large screen, an MQTT Broker agent, a web front-end server, an internal network, and the like.

[0089] Further, the automatic test monitoring system of the embodiment is composed of fourteen parts: an automatic tool, a wireless network card control windows service, an upgrade server, a high and low temperature box API server, a web background server, a file management server, a database management server, a time calculation service, a short message server, a log server, a monitoring large screen, an MQTT Broker agent, a web front-end server, and an internal network.

[0090] 1. Automatic tool

[0091] Any programming language can be used for development. The program is responsible for implementing the business logic of reliability, including controlling the start of the temperature box, setting the target temperature and humidity, controlling the flow test equipment to flow to the device under test, wireless flow test, starting multiple background tasks during all business tests, reading the real-time humidity and temperature of the temperature box at intervals, reading the output power of the AC or DC source, reading the monitored power supply voltage of the voice fixture, reading the voltage of the USB load, reading the real-time throughput of the wireless flow, reading the real-time packet loss of the wired flow, reading the real-time temperature of each channel configured by the temperature acquisition instrument, reading the power of each channel of the POE load, reading the uplink and downlink rates of multiple remote wireless network cards, and the like. The above are the reading of the supported monitoring indicators. Which indicators should be monitored in actual testing can be selected by the tester. The data read at intervals will be published through the MQTT protocol through a specific topic.

[0092] 2. Wireless network card control windows service

[0093] A windows service installed on a remote computer, which starts with the computer and can be called by any automatic tool through RPC to complete the control of the network card after the service is started. In the present application, the method of reading the uplink and downlink rates of the wireless network card is mainly called.

[0094] 3. Upgrade server

[0095] Provide upgrade service for automation tools, because automation tools are deployed in different stations, version control management is difficult, so an upgrade server is needed, when the automation tool starts, it submits a query to the upgrade server to check if there is a new version of the tool, if there is, it will upgrade.

[0096] 4. High and low temperature box API server

[0097] Responsible for providing various software interfaces for operating the automation tool to provide the temperature box.

[0098] 5. Web background server

[0099] Web background server is a server that provides similar login authentication, session management, processing of various data submitted by web front-end, and visualization of various data in the database for web front-end.

[0100] 6. Web front-end server

[0101] Mainly provides static files for user browsers, html, css, javascript pages, when the user requests, the server transmits static files to the browser, and after the browser renders, it loads data according to the control logic in Javascript, and provides some interfaces for users to add, delete, modify and query.

[0102] 7. File management server

[0103] Used to receive submission, save, download and other functions of result files from automation tools;

[0104] 8. Database management server

[0105] Used to subscribe to related interfaces, process messages sent by different MQTT topics, and store data in a certain format into the database

[0106] 9. Time calculation service

[0107] This service is best deployed separately from other services, as this service needs to have a public network environment; its role is to crawl all non-working days of the year on the first day of the year and store them into the database; it provides interfaces to other services, other services send a start time and an end time, it will calculate the total duration of this time period, non-working time duration (non-working days are calculated as 24 hours, working days are calculated as all time except working hours), daily duration, and then return the result to the corresponding service in a certain format.

[0108] 10. SMS server

[0109] This service is best deployed separately from other services when deployed, because this service needs to have a public network environment; its role is to connect to the cloud server of a specific SMS service provider, and provide a specific SMS interface to send SMS.

[0110] 11、Log Server

[0111] It monitors the specific topic of SMS, extracts the content, time, and level of the sent log, and then stores it.

[0112] 12、Monitor Big Screen

[0113] To show the running projects, workstation occupancy, and real-time status of each temperature box in the current system, and display some indicators in the form of visual charts.

[0114] 13、MQTT Broker Agent

[0115] It plays a coordinating role as an agent, an automation tool, a monitor big screen front end, a web front end, a web back end, and other services will be connected to this agent service through MQTT; it will distribute specific topic messages to each subscriber.

[0116] 14、Internal Network

[0117] The carrier of the message; the transmission of messages between the above modules must pass through the internal network, so this network must ensure stability.

[0118] II. Implementation method and effect of part of the technical features in the embodiment

[0119] 1、Automation Tool

[0120] Any programming language can be used for development, and the program is responsible for implementing the business logic of reliability, including controlling the start of the temperature box, setting the target temperature and humidity, controlling the flow test device to flow to the device under test, wireless flow testing, starting multiple background tasks during all business tests, reading the real-time humidity and real-time temperature of the temperature box at intervals, reading the output power of the AC or DC source, reading the monitored power supply voltage of the voice tool, reading the voltage of the USB load, reading the real-time throughput of wireless flow, reading the real-time packet loss of wired flow, reading the real-time temperature of each channel configured by the temperature acquisition instrument, reading the power of each channel of the POE load, and reading the uplink and downlink rates of multiple remote wireless network cards. The above are the supported monitoring index readings, and which indexes should be monitored in actual testing can be selected by the tester. The data read at intervals will be published through the MQTT protocol through a specific topic. Its test log, test progress, and station progress will also be sent out through a specific MQTT topic after the test starts. In this way, the tester can know the current test situation remotely after the test starts.

[0121] 2. Wireless network card control windows service

[0122] The feature is a windows service installed on a remote computer, which starts with the computer and can be called by any automation tool through RPC to control the network card after the service is started. In the invention, the method of reading the uplink and downlink rates of the wireless network card is mainly called. It also provides methods such as network card connection, which can be called through RPC.

[0123] 3. Upgrade server

[0124] The upgrade server provides upgrade services for automation tools. Because automation tools are deployed in different stations, version control management is difficult, so an upgrade server is needed. The automation tool submits to query the upgrade server whether there is a new version when it starts, and upgrades if there is.

[0125] 4. Web background server

[0126] After receiving the data submitted by the front end (GET, POST), it will verify whether the submitted parameters are legal and whether there are missing according to the model class defined by each interface. Then it will also verify whether the processed data is legal and whether there are missing according to the return value model class defined by each interface, to ensure that the data submitted by the front end will not affect the security and stability of the system, and will not cause the front end to be abnormal because of the abnormal return data of the back end, to ensure compliance on both sides.

[0127] 5. web front-end server

[0128] 51. Service status page: After the page is loaded, it first queries all the project task information created by the automation tool on the server at the beginning of the test; a table is formed; each row of the table contains the basic information of each project task, such as creation time, work area, workbench, project name, task name, and other basic information, and several columns of dynamic parameters are left blank, such as data file size, current state, current progress, real-time log, and several columns; there is also a management column; after the table is loaded, the corresponding topic is directly subscribed in the Javascript front-end logic; when the automation client sends the current state and current progress messages, the two states are updated in real time; so that the business can complete the two-way communication of data without overall or partial refresh, and the execution state of the automation client can be directly seen on the page display; if the current state is updated to running, a button will be updated in the real-time log cell, and clicking the button will subscribe to a specific topic; the log information sent by the automation tool through the topic will be displayed in the opened modal box; if it is an Error level log, the background is marked in red, and the Warring level is marked in yellow; of course, the background color is only used to distinguish different levels of logs, and it is enough to distinguish; the specific color is not a feature, and the color distinction is the feature of the present application. A monitoring data button will appear in the management cell, and clicking the button will appear a modal box; the modal box will appear a button for temperature box, AC source, DC source, wired throughput, wireless throughput, wireless network card rate, POE, temperature collection, VOIP power supply, USB voltage, etc. when initialized; clicking the corresponding button will appear the corresponding index, which is a curve with time as the X axis and the measured value of each index as the Y axis; the test progress cell will present the current execution progress in the form of a progress bar; if the test is completed, a download file button will appear, and clicking the file will download the compressed package composed of screenshots generated during the test.

[0129] 52. Workbench management page: there is an add workbench button, and clicking the button will appear an interface for adding workbench information; the workbench information added by the user here will be synchronized to the workbench selection box of the automation client. There is a table to display all the workbench information configured in the current database; each workbench management cell has a project information button, through which the workbench can be manually bound to a certain project; if it is manually bound to a certain project, the drop-down box of the automation client cannot be selected; when the project is bound, the planned start time and end time of the project, user information, etc. will be indicated. If it is an automation execution workbench, the current workbench execution progress will be presented in the form of a progress bar.

[0130] 53. The incubator management page: there will be an increase in the incubator button, click the button will appear to increase the incubator information interface, the user through the incubator information added here will be synchronized to the incubator selection box of the automation client. In the form of a small card to present the current state of all incubators in the database: real-time temperature, real-time humidity, set temperature, set humidity, whether there is an alarm, the current who occupies the information; In the form of a table to present all the online records of the incubators in the database, record usage.

[0131] 6. File management server

[0132] For receiving the submission of result files from automation tools, saving, downloading, etc.

[0133] 7. Database management server

[0134] For subscribing to related interfaces, processing messages sent by different MQTT topics, and storing data in a certain format into the database

[0135] 8. Time calculation service

[0136] The service is preferably deployed separately from other services when deployed, as this service needs to have a public network environment; its role is to crawl all non-working days of the year from a specific website on the first day of the year and store them in the database; it provides an interface to other services, and other services send a start time and an end time, which will calculate the total duration of the time period, the non-working time duration (non-working days are calculated as 24 hours a day, and working days are calculated as all times except the working time), and the daily duration, and then return the result to the corresponding service in a certain format.

[0137] 9. SMS server

[0138] The service is preferably deployed separately from other services when deployed, as this service needs to have a public network environment; its role is to connect to the cloud server of a specific SMS service provider, and provide a specific SMS interface to send SMS; the feature of the present invention is that after the automation tool test is completed, the content of the SMS and the contact information will be sent through a specific topic of MQTT; the SMS server will subscribe to this topic, and when it listens to the topic, it will parse the message content and call the test completion SMS template to send the SMS. It will also listen to the incubator alarm SMS topic, and when the incubator service module listens to the incubator alarm, it will call the specific SMS template to tell the specific contact about the abnormality of the incubator.

[0139] 10. Log server

[0140] It monitors the specific topic of SMS, extracts the content, time, level of the sent log, and then stores it. The feature of the application is that the server provides a specific interface, which returns all logs within one hour of the data time point to the front end when the user clicks on the abnormal data point of the specific curve of the front end.

[0141] 11、Monitor large screen

[0142] It is used to show the running projects in the current system, the station occupation, the real-time state of each temperature box, and some indicators in the form of visual charts. The feature of the application is that it is different from the conventional BI large screen in that it connects the background API or the background database in a timed manner, and presents the processed information to the interface. The difference is that it first loads the basic information, such as the basic static information of the temperature box, and then subscribes to a specific topic in the front-end Javascript. Once the information of the topic changes, the interface will change directly. There is no need to request repeatedly.

[0143] The beneficial effects of the application are embodied in the above-mentioned preferred embodiments of the application, and are not intended to limit the application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. An automated test monitoring method, characterized by, The steps of the test monitoring method include: Implementing the business logic of environmental reliability testing through an automated tool; Starting multiple background tasks, reading and recording test indicator data at preset intervals; Publishing the indicator data to a specific topic through the MQTT protocol; Processing and analyzing the test indicator data; Visualizing the processed test indicator data.

2. The automated test monitoring method of claim 1, wherein, The steps of implementing the business logic of reliability testing through an automated tool include: Initializing the test environment, including setting the parameters of the temperature and humidity chamber; Configuring the flow test equipment; Starting the device under test and performing initialization configuration.

3. The automated test monitoring method of claim 1, wherein, The steps of starting multiple background tasks, reading and recording test indicator data at preset intervals include: Reading temperature and humidity data; Reading power output, feed voltage, and USB voltage; Reading wireless flow throughput and wired flow packet loss rate; Reading channel temperature and POE load power; Reading remote wireless network card speed.

4. The automated test monitoring method of claim 1, wherein, The steps of publishing the indicator data to a specific topic through the MQTT protocol include: Formatting the read indicator data into unified JSON data; Designing specific MQTT topics for different types of indicator data; Connecting the automated tool to the MQTT Broker agent; Using the publish method of MQTT to publish the JSON data to the corresponding topic; Setting appropriate quality of service levels according to data importance.

5. The automated test monitoring method of claim 1, wherein, The steps of processing and analyzing the test indicator data include: Cleaning and preprocessing the raw data; Calculating performance indicators; Performing anomaly detection and alerting.

6. The automated test monitoring method of claim 5, wherein, The formula of the calculation performance index is: Where T is the current temperature, Tmax is the maximum allowed temperature; P is the current power, Pmax is the maximum allowed power; L is the current packet loss rate, Lmax is the maximum allowed packet loss rate; S is the current throughput, Smax is the maximum throughput; w1, w2, w3, w4 are weight coefficients.

7. An automated test monitoring system for implementing the automated test monitoring method according to any one of claims 1 to 6, characterized in that The test monitoring system includes: An automated tool for implementing the business logic of reliability testing; An MQTT Broker agent for managing the publication and subscription of MQTT messages; A web backend server for processing and analyzing test data; A web frontend server for providing a user interface and data visualization; and A database management server for storing and managing test data. The automated tool includes: A temperature and humidity control module for controlling the temperature and humidity chamber; A flow test module for controlling the flow test equipment; A device configuration module for configuring the device under test.

8. The automated test monitoring system of claim 7, wherein, The test monitoring system further includes: A wireless network card control Windows service for reading the uplink and downlink rates of the remote wireless network card; An upgrade server for version management and upgrade of the automated tool; A high and low temperature chamber API server for providing a software interface for operating the temperature chamber; A file management server for managing files generated during the test process; A time calculation service for processing time-related data; An SMS server for sending alert information; A log server for recording system logs; A monitoring large screen for displaying the test status in real time; and An internal network for connecting various components of the system.

9. The automated test monitoring system of claim 7, wherein, The Web background server includes: a data processing module for data cleaning and preprocessing; an index calculation module for calculating key performance indicators; and an anomaly detection module for detecting anomalies and triggering alarms.

10. The automated test monitoring system of claim 7, wherein, The Web front-end server includes: a data visualization module for generating charts and dashboards; a user interaction module for processing user input and operations; a real-time update module for real-time updating of display data.

Images