Dynamic performance evaluation and test method and system for mixed access scene of Internet of Things equipment

Abstract

The invention provides a dynamic performance evaluation and test method and system for an Internet of Things equipment hybrid access scene, and the method comprises the steps: setting equipment simulators of a plurality of Internet of Things services, and configuring key service parameters; generating and reporting service data based on the key service parameters; obtaining business data flow information based on the business data storage condition; comparing the service data flow information with a service data control threshold interval to generate an adjustment request; and according to the adjustment request, adjusting the configuration of the equipment simulator of the corresponding type, and updating the configured service parameters. According to the application, the service data reported by the simulation device can be adjusted in time according to the service expected change models corresponding to different Internet of Things devices, and the risk assessment of the bearing capacity of the Internet of Things platform under the condition of service fluctuation is more targeted and predictable.

Description

Technical Field

[0001] The present application belongs to the field of system testing technology and relates to a performance evaluation method, and in particular to a dynamic performance evaluation and testing method and system for hybrid access scenarios of IoT devices. Background Art

[0002] In the field of system testing, in order to ensure that the system functions normally under some large-capacity and large-business flow conditions, performance evaluation tests for given scenarios will be carried out before delivery, and relevant performance parameters such as system capacity and response speed will be provided to facilitate the smooth implementation of the system.

[0003] At present, the general system performance evaluation scheme is mainly based on the performance and capacity evaluation under the fixed growth load model. In other words, it is generally based on the number of online devices in a certain number of systems, the number of business requests that grow smoothly or step by step, and the specific business request peak and duration. It can also evaluate the performance bottleneck and capacity limit of some services in the system to a certain extent. It has a wide range of applications in B / S architecture systems with a large number of http requests.

[0004] However, in the actual use scenarios of IoT systems, the mixed use of multiple IoT devices will cause the corresponding actual number of online devices and the amount of reported business data and requests to be affected by factors such as network connection, business processing delay, and data batch processing / transmission, resulting in dynamic fluctuations in random business growth, and even pulsed growth and extremely fast decline in certain time periods. These dynamic fluctuations will cause the system to have performance bottlenecks and instability in back-end services beyond the capacity limit, resulting in abnormal business processing, jams, or unreasonable occupation of system resources and the inability to quickly recover and respond. Once these problems occur on site, due to the complexity of IoT device installation and mixed business scenarios, additional problem location and processing costs will be introduced, affecting the smooth delivery and operation of the system.

[0005] Therefore, before the system is delivered to users, how to expose the system's processing vulnerabilities under business load fluctuations as much as possible before delivery through business evaluation methods that are closer to real IoT business demand scenarios, and improve the robustness of the IoT service system in different business scenarios, has become a technical difficulty that technical personnel need to urgently solve. Summary of the invention

[0006] The purpose of this application is to provide a dynamic performance evaluation and testing method and system for hybrid access scenarios of IoT devices, which is used to solve the problem in the prior art that due to the mixed use of multiple IoT devices, the application of the hybrid system is affected by multiple factors, as well as the dynamic fluctuations of random business growth, resulting in performance bottlenecks and instability of back-end services beyond the capacity limit.

[0007] In the first aspect, the present application provides a dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices, 1. Applied to multiple IoT systems, the method includes: setting up device simulators for multiple IoT services and configuring key business parameters; generating and reporting business data based on the key business parameters; obtaining business data flow information based on the business data entry status; comparing the business data flow information with the business data control threshold interval to generate an adjustment request; according to the adjustment request, adjusting the configuration of the corresponding type of device simulator, and updating the configured business parameters.

[0008] In an implementation method of the first aspect, device simulators for multiple IoT services are set up, and key business parameters are configured, including: corresponding device simulators are set up according to multiple business system devices and the protocols used by the business system devices; the device simulators include: a pass device simulator, a sensing device simulator, and a smart screen device simulator; according to business needs, business parameter configuration is performed on the device simulator; wherein the key business parameters include: device attributes, reporting period, reporting frequency, and reported business data / format.

[0009] In an implementation method of the first aspect, the business data includes: traffic data, sensing data, and smart screen status data; generating and reporting business data based on the key business parameters includes: formulating data reporting rules according to business needs; designing data generation logic based on the key business parameters and the data reporting rules; and generating simulated business data according to the data generation logic.

[0010] In an implementation of the first aspect, the business data flow information is compared with the business data control threshold interval, and the adjustment request is generated, including: setting each business data control threshold according to business needs; comparing the business data flow information with the business data control threshold to determine whether the business data flow information meets the trigger condition; when the business data meets the trigger condition, generating an adjustment request.

[0011] In an implementation of the first aspect, the business data flow information is compared with the business data control threshold to determine whether the business data meets the trigger conditions, including: when the business data flow information exceeds the upper and lower limits of the business data control threshold, it is judged to be an abnormal state, then an alarm mechanism is triggered and an adjustment request is generated; when the business data flow information is within the upper and lower limit interval of the business data control threshold, it is judged to be a normal state, then the alarm mechanism is not triggered.

[0012] In an implementation of the first aspect, it also includes: before the business data is reported in batches, starting the business control module, and reporting the business data of the corresponding type according to the initial business configuration parameters or the business configuration parameters adjusted after the adjustment request is triggered.

[0013] In an implementation of the first aspect, according to the adjustment request, adjusting the configuration of the corresponding type of device simulator includes: calculating the configuration amount to be adjusted according to the adjustment request, the business data control threshold and the expected limit of business growth; adjusting the configuration of the business system device simulator according to the adjustment amount; wherein the adjustment content includes: data volume, frequency, and duration.

[0014] In an implementation of the first aspect, it also includes: after completing the adjustment of the configuration of the device simulator, restarting the device simulator and performing a validation; using the updated business parameter configuration and reporting the new business data.

[0015] On the second aspect, the present application provides a dynamic performance evaluation and testing system for hybrid access scenarios of IoT devices, including: a simulator setting module, used to set up device simulators for multiple IoT services and configure key business parameters; a data generation module, used to generate business data based on the key business parameters; a data warehousing and monitoring module, used to obtain business data flow information based on the business data warehousing situation; a data comparison module, used to compare the business data flow information with the business data control threshold interval to generate an adjustment request; a business control module, used to adjust the configuration of the corresponding type of device simulator according to the adjustment request, and update the configured business parameters.

[0016] In the last aspect, the present application provides an electronic terminal for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices, comprising: a processor and a memory. The memory is used to store a computer program; the processor is connected to the memory and is used to execute the computer program stored in the memory, so that the dynamic performance evaluation and testing device for hybrid access scenarios of IoT devices executes the dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices.

[0017] As described above, the method and system for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices of the present application have the following beneficial effects:

[0018] The present application provides a dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices. In the process of mixed IoT service performance evaluation, a mechanism for real-time monitoring and dynamic linkage adjustment for different IoT services is added. The mechanism is mainly achieved by linking the monitoring module, the control module, and the simulator during the stress test to obtain the service thresholds and dynamic durations of IoT services corresponding to different services set according to the business growth model based on actual business data, and to perform controllable dynamic load evaluation on IoT services / backends. Compared with the current fixed business volume or step-shaped growth load simulation method, the solution provided by the present application can timely adjust the business data reported by the simulation device according to the expected business change model corresponding to different IoT devices, and is more targeted and predictable in the risk assessment of the carrying capacity of the IoT platform under business fluctuations. The method provided by the present application has a wide range of applications and high utilization value. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Shown is a schematic diagram of a hardware application scenario in one embodiment of the dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices described in this application.

[0020] Figure 2 Shown is a schematic diagram of the overall process of a dynamic performance evaluation and testing method for a hybrid access scenario of IoT devices described in this application in one embodiment.

[0021] Figure 3 Shown is a business process diagram of an embodiment of the dynamic performance evaluation and testing method for the hybrid access scenario of IoT devices described in this application.

[0022] Figure 4 Shown is a schematic diagram of module interaction and data flow in one embodiment of the dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices described in this application.

[0023] Figure 5 Shown is a flow chart of S2 in the dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices described in this application.

[0024] Figure 6 Shown is a flow chart of S4 in the dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices described in this application.

[0025] Figure 7 Shown is a flow chart of S5 in the dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices described in this application.

[0026] Figure 8 Shown is a schematic diagram of the curve of the existing tool step-by-step pressurization method.

[0027] Fig. 9 Shown is a schematic diagram of the dynamic linkage pressure adjustment and IoT business volume curve in the dynamic performance evaluation and testing method for the hybrid access scenario of IoT devices described in this application.

[0028] Fig.10 Shown is a graph of employee entry and exit statistics in one embodiment of a dynamic performance evaluation and testing method for a hybrid access scenario of IoT devices described in this application.

[0029] Fig.11 Shown is a schematic diagram of the principle structure of a dynamic performance evaluation and testing device for a hybrid access scenario of IoT devices described in this application in one embodiment.

[0030] Fig.12 Shown is a schematic diagram of the principle structure of an electronic terminal for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices described in this application in one embodiment.

[0031] Component number description

[0032] 11 Simulator

[0033] 12 Business data reporting module

[0034] 13. Traffic monitoring module

[0035] 14 Business Control Module

[0036] 111 Simulator Settings Module

[0037] 112 Data generation module

[0038] 113 Data storage and monitoring module

[0039] 114 Data comparison module

[0040] 115 Business Control Module

[0041] 121 Processor

[0042] 122 Memory DETAILED DESCRIPTION

[0043] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0044] It should be noted that the illustrations provided in the following embodiments are only used to illustrate the basic concept of the present application in a schematic manner, and therefore the illustrations only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0045] The following embodiments of the present application provide a dynamic performance evaluation and testing method and system for a hybrid access scenario of IoT devices, which solves the problem in the prior art that due to the mixed use of multiple IoT devices, the application of the hybrid system is affected by multiple factors, as well as the dynamic fluctuations of random business growth, resulting in performance bottlenecks and instability of back-end services beyond the capacity limit.

[0046] like Figure 1 As shown, the hardware application scenario schematic diagram of the dynamic performance evaluation and testing method for the hybrid access scenario of the Internet of Things devices described in the present application in one embodiment specifically includes: a simulator, a business data reporting module, a business data warehousing / traffic monitoring module, and a business data control module. Among them, corresponding simulators are designed for the main business-bearing traffic equipment, sensing equipment, and smart screen equipment in the current Internet of Things business docking system. These simulators all open configurable key business setting parameters and dock with the business data reporting control module. The business data reporting module reads the warehousing status of the corresponding business data in the database in real time, and reports the adjustment request to the business control module in a timely manner in combination with the control thresholds of various services that have been set. According to the adjustment request reported by the monitoring module, combined with the configuration control threshold and the expected upper and lower limits of business growth, the automatic adjustment of the simulator configuration of the relevant equipment is completed, and the simulator is restarted after the adjustment is completed to ensure that the simulator uses the new business parameter configuration to report the business data. This application is aimed at the situation that the current general performance evaluation scheme is unable to accurately evaluate the mixed load and fluctuation of IoT services in real scenarios, and introduces a test scheme for dynamic evaluation of IoT service system performance; based on the current mixed access of IoT devices and the different interactive performance of different types of devices, network transmission / network load and business response fluctuations, as well as the actual business situation where the efficiency of IoT service processing is prone to random fluctuations, a corresponding performance evaluation scheme is proposed.

[0047] The following will describe in detail the dynamic performance evaluation and testing method for the hybrid access scenario of IoT devices provided in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application.

[0048] See also Figures 2 to 4 .like Figures 2 to 4 As shown, this embodiment provides a dynamic performance evaluation and testing method for a hybrid access scenario of IoT devices.

[0049] The dynamic performance evaluation and testing method of the IoT device hybrid access scenario specifically includes the following steps:

[0050] S1, set up device simulators for various IoT services and configure key service parameters.

[0051] S11, setting corresponding business parameters according to a variety of different business system devices and the protocols used by the business system devices, and obtaining device simulators corresponding to each business system; the device simulators include but are not limited to: a traffic device simulator, a sensing device simulator, a smart screen device simulator, etc.;

[0052] S12, configuring service parameters for each of the device simulators according to service requirements. The key service parameters include but are not limited to: device attributes, reporting cycle, reporting frequency, reporting service data / format, etc.

[0053] In this embodiment, corresponding simulators are designed for the main business-carrying access devices, sensing devices, and smart screen devices in the current IoT business docking system. These simulators all open configurable key business setting parameters to facilitate subsequent docking with the business control module.

[0054] Specifically, according to the main devices used in the IoT business docking system, such as access devices, sensing devices, smart screen devices, etc., select the corresponding simulator (such as access device simulator, sensing device simulator, smart screen device simulator). According to business needs, configure the key business parameters of each simulator, such as IoT device ID, device type, communication protocol, device attributes, data reporting cycle, data reporting frequency, specific collected data (such as face, QR code, card number, personnel status, smart screen status) and timestamp, etc., to ensure that the configuration parameters are consistent with the IoT business docking system.

[0055] For example, the sensing device is used to sense and monitor whether there is someone at the employee's seat; the sensing device here can be a sensor, infrared, microwave, radar and other detection equipment, and is also used in conference rooms, offices and other places where employees enter or are in place. Further, establish a communication connection, and configure the communication parameters with the business data reporting control module in the simulator, such as: IP address, port number, communication protocol, etc. According to the requirements of the business data reporting module, configure the format of the data reported by each simulator.

[0056] Specifically, first, according to the devices in various IoT business docking systems, determine the key factors such as the role, communication protocol, data format, and control instructions of each device in the business process. According to business needs, plan the core functions of the simulator; design the overall architecture of the simulator, such as: data processing module, communication module, control module, user data module, etc., to determine the communication interface and interaction protocol between each device simulator and the IoT business docking system. Here, an interface is used to send a configuration to the device simulator, and the device simulator will be modified according to this configuration and then restarted.

[0057] Then, write simulator code to realize data processing, communication control and other functions. For example: develop simulation functions such as face recognition / code scanning / card swiping result feedback / transmission for the access device simulator; develop simulation functions such as status information feedback / transmission for the sensing device simulator; develop simulation functions such as online status, display data transmission interaction, etc. for the smart screen device simulator.

[0058] Next, each device simulator is tested to ensure that the module functions and communication functions are normal.

[0059] It should be noted that not only access equipment (such as cameras, access control, panel machines, etc.), sensing equipment (such as temperature sensors, humidity sensors, infrared instruments, microwave detectors, radars and other detection equipment), and smart screen equipment (such as touch screen devices, billboard display devices, status display devices, information display devices, etc.) are used here. These are commonly used equipment. Other corresponding equipment can be added in specific occasions or when there are specific needs.

[0060] S2, generating business data based on the key business parameters. Figure 5 , which is a schematic diagram of the process of S2 in the table image matching method described in this application. Figure 5 As shown, S2 comprises the following steps:

[0061] S21, formulate data reporting rules according to business needs;

[0062] S22, designing data generation logic based on the key business parameters and the data reporting rules;

[0063] S23, generating simulated business data according to the data generation logic.

[0064] In this embodiment, first, the communication parameters required for business data reporting (such as IP address, port number, communication protocol, etc.) are configured in the business system device simulator (such as a traffic device simulator, a sensing device simulator, a smart screen device simulator, etc.), and each simulator is started to establish a communication connection relationship with the business data reporting.

[0065] Further, according to the business data reporting requirements, configure the format of the reported data of the business system device simulator. Then, simulate the device operating status in each device simulator and generate corresponding business data. Next, convert the business data according to the configured data format and report it to the business data control module through a communication connection.

[0066] Specifically, first, identify the key business parameters in the business system, such as: device ID, sensor type, data value, time, etc. Then, according to the business requirements, formulate data reporting rules, including: reporting frequency, data format, data verification, etc. Next, design the data generation logic based on the key business parameters and data reporting rules. This can include randomly generating data, generating data based on historical data, or generating data according to preset scenarios. Write a data generation script using a programming language (such as: Python, Java, nodejs, etc.) to implement the data generation logic. Then run the data generation script to generate simulated business data and verify the accuracy and business relevance of the data. Finally, according to the test results, adjust and optimize the data generation script to ensure that the generated simulated data meets the business requirements and data reporting rules. Report the generated simulated data to the background service through the defined interface specification to drive subsequent processing and analysis.

[0067] For example, the reporting rule here can be understood as: assuming that M data should be uploaded per minute, then it will be monitored to query whether there are really M data in a certain time period, and sampling detection can be used for processing; if there are not M data, the frequency will be modified accordingly. At the same time, if there are M data in the previous time period and the data volume is to be increased to N (M < N) or more in the subsequent time period, then according to the user's needs, it can be modified at the whole hour or after a given time interval. It should be noted that the data volume here depends on the user's needs.

[0068] S3. Obtain the business data traffic information based on the business data storage situation. Specifically, it includes the following steps:

[0069] S31. Obtain the business data storage information within a certain time period;

[0070] S32. Obtain the business data traffic information based on the actual business data storage information.

[0071] In this embodiment, according to the business requirements, write a database reading program or script to regularly or real-time obtain the business data storage information in the database. Then, use a network traffic monitoring algorithm to obtain the traffic information of the business data.

[0072] Specifically, according to the requirements, determine the types of business data to be monitored and the storage situation; and determine the control thresholds for each business data type, such as: data storage volume, traffic peak, etc.

[0073] It should be noted that the business data flow information here refers to the business data storage situation and flow data, such as: how many data flows are stored per second, which is a process of calculating the number of data, and which time periods have the most data flow.

[0074] S4, comparing the service data flow information with the service data control threshold interval, and generating an adjustment request. Figure 6 , which is a schematic diagram of the process of S4 in the table image matching method described in this application. Figure 6 As shown, S4 comprises the following steps:

[0075] S41, setting the control threshold of each business data according to business requirements;

[0076] S42, comparing the service data flow information with the service data control threshold to determine whether the service data reaches a trigger condition;

[0077] S43: When the service data flow information reaches a trigger condition, an adjustment request is generated.

[0078] In this embodiment, by reading the storage status of the corresponding business data in the database in real time and combining the control thresholds of various services that have been set, the adjustment request is reported to the business control module in a timely manner.

[0079] When the business data flow information is not equal to the business data control threshold and exceeds the set upper and lower limits, it is judged to be an abnormal state, then the alarm mechanism is triggered and an adjustment request is generated; when the business data flow information is equal to the business data control threshold or is within the upper and lower limits, it is judged to be a normal state, then the alarm mechanism is not triggered.

[0080] Specifically, the business data storage situation and traffic data are judged according to the set control threshold. That is, when the data traffic exceeds the control threshold range (upper and lower limits), the adjustment request reporting logic is triggered; the adjustment request is sent to the business control module. The adjustment request should contain necessary information, such as data type, threshold, current data situation, etc.

[0081] That is to say, according to the monitoring purpose and needs, set comparison rules to compare the collected storage information and business data flow information.

[0082] S5, according to the adjustment request, adjust the configuration of the corresponding type of business system device simulator, and update the configured business parameters. Figure 7 , which is a flow chart of S5 in the table image matching method described in this application. Figure 7 As shown, S5 comprises the following steps:

[0083] S51, calculating the configuration amount to be adjusted according to the adjustment request, the service data control threshold and the expected service growth limit;

[0084] S52, adjusting the configuration of the business system device simulator according to the adjustment amount; wherein the adjustment content includes but is not limited to: data volume, frequency, duration, etc.;

[0085] S53, after completing the adjustment of the configuration of the device simulator, restart the device simulator and perform a validation;

[0086] S54, using the updated service parameter configuration to report the new service data.

[0087] In this embodiment, a business control module is added to automatically adjust the simulator configuration of related equipment based on the adjustment request reported by the monitoring module in combination with the configuration threshold and the expected upper and lower limits of business growth. After the adjustment is completed, the device simulator is restarted to ensure that the device simulator uses the new business parameter configuration to report business data.

[0088] Specifically, first, the data flow monitoring result reported by the monitoring module is received.

[0089] Then, based on the reported business data traffic results and the configured thresholds and the upper and lower limits of expected business growth, it is determined whether the data traffic exceeds the threshold range or does not meet the business growth expectations. If it exceeds, a downward adjustment is made, and if it is insufficient, an upward adjustment is made. The corresponding adjustment request is generated according to the rules; if no adjustment is required, no adjustment request is generated.

[0090] Next, according to the parameter information in the adjustment request, write code to automatically adjust the simulator business configuration of the relevant device to ensure that the adjusted configuration meets the business needs and system requirements. Restart the device simulator and ensure that the simulator can use the new business parameter configuration to report business data.

[0091] Finally, conduct a test evaluation. That is, integrate the business control module into the existing system to ensure normal communication and collaboration with the monitoring module, simulator and other components. Perform a comprehensive test on the integrated system to ensure that the interfaces and data transmission between modules are correct. Then, perform interface tests and comprehensive tests on the business control module to ensure that it can correctly receive and process adjustment requests, automatically adjust simulator configurations and restart the simulator. Test the module's performance in different scenarios, such as high concurrency, exception handling, etc.

[0092] It should be noted that the business control module also needs to archive the business adjustment results in real time to provide the business data stress test adjustment time record and business model data basis for subsequent test analysis.

[0093] See also Figures 8 to 10 , respectively showing a schematic diagram of the step-by-step pressurization curve of the existing tool, a schematic diagram of the dynamic linkage pressurization adjustment and IoT business volume curve in the dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices described in this application, and an employee entry and exit statistics curve in one embodiment of the dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices described in this application.

[0094] The following takes the dynamic linkage adjustment process of the stress testing process between modules as an example to illustrate.

[0095] First, set up the setting simulators corresponding to the devices of various IoT business systems, refine the initial key parameter configuration according to business needs, and apply it in the startup of the multi-protocol device simulator, so as to perform bearer push simulation of IoT device business data of various different protocols (such as: MQTT, Bluetooth, HTTP, etc.) reported to the back-end server.

[0096] Then, before the business data is pushed, start the business data reporting monitoring module and view the results of real-time or sampling monitoring of key business data and traffic reported to the back-end server during the performance test.

[0097] For example, the monitoring judgment standard here can be understood as follows: assuming that 20 data should be uploaded every minute, then monitoring will be used to check whether there are really 20 data in a certain period of time, and sampling detection can be used for processing; if there are not 20 data, the frequency will be modified accordingly. Fig. 9 If there are 20 data in the first 30 minutes and you want to increase the amount of data to 30 or 50 or more in the last 30 minutes, you can modify it at the hour or after a given time interval according to user needs. It should be noted that the amount of data here depends on the actual user needs.

[0098] Before the batch reporting of business data, the business data control module is started and the business data reporting monitoring results are received. When the control threshold range of some or all businesses is reached, the related business data reporting content, such as data volume, frequency, duration and other key parameters, is triggered for dynamic adjustment. At the same time, before the batch reporting of business data, the configuration control module of business data interaction is started. When the dynamic adjustment of business key parameters is triggered, the key parameters of the business model such as the corresponding type of business data reporting content, data volume, frequency, duration and so on are configured and adjusted.

[0099] Finally, after the business data reporting configuration is adjusted, confirm the adjustment results of the reporting configuration of the simulation business module, and check the adjustment time point and the results summarized in the archive of the control module.

[0100] In summary, combined with Figure 8 and Fig. 9 It can be seen that, unlike traditional servers, if the traditional pressurization method is slow in processing, data will accumulate, and it will present a step-by-step rising process. It can be seen from the IoT business control curve in the figure that, for example, when employees enter their clock-in information, in general, data from the peak hours of morning, noon, and evening will be collected. Here, we can see that there will be an increase every 5 minutes, with more concentrated clock-ins or passages. After 5 minutes, it will fall back, and after another 5 minutes, it will rise again, and it will change continuously in this way. The method used in this application is for the situation of employees in an area (or: a point), which can include the situation of many employees.

[0101] Therefore, it can be seen from the pressure curve that the dynamic adjustment solution has greater adaptability to the business bearer drive under the hybrid business model.

[0102] The following uses the control process of dynamic business adjustment as an example to illustrate.

[0103] First, the initial values ​​of data reporting for different services have been set, and it is confirmed that the batch business data generated by the multi-protocol simulator has created a large business load pressure on the IoT service background through the collection point control method.

[0104] For example, the main reports include: the number of virtual devices, frequency, random status or face capture results, QR code scanning feedback, card number, etc. The initial value here mainly refers to the number of devices in the system and the capacity value. For example, if there are 100 panel machines, a fixed frequency needs to be set (it is expected that each machine will report 2 data per second, and the amount of data that needs to be processed per second in the background is also 2*100=200).

[0105] During this process, the business data flow monitoring module will continue to monitor the actual background business data storage situation. If a certain type of business data reporting (storage) is not within the upper and lower limits of the threshold, it will trigger an adjustment request to the business data control module.

[0106] Then, after receiving the adjustment request, the business data control module adjusts the corresponding business simulator data reporting configuration according to the business growth pattern set in the system (such as: the peak value, average data volume, duration, etc. of a certain business that needs to be processed within a unit time period); and restarts the corresponding simulator to confirm that the simulator uses the new configuration to re-report the corresponding business data, thereby generating service pressure on the server that is more in line with business requirements.

[0107] Finally, after the adjustment process is completed, the control process data is archived and key business performance indicators are continuously collected simultaneously.

[0108] Embodiment 1: Take the mixed test of face recognition service and code scanning service as an example for explanation.

[0109] During the mixed testing of face recognition and code scanning services, the face recognition processing link does not need to dynamically generate QR codes, so the same business concurrency can reach the set threshold in a shorter time. However, after the code scanning service goes through the steps of login-querying personnel information-dynamic generation of QR codes, it cannot reach the set threshold in a corresponding multiple of time under a given concurrency. At this time, the existing performance testing tools need to perform distributed processing or suspend existing stress testing to increase concurrency to manually adjust the stress testing parameters. The dynamic adjustment solution, after monitoring that the data entry of the code scanning service is lower than the expected control threshold, will quickly reach the expected threshold of the set service load by automatically adjusting the service interval time or increasing the number of simulators.

[0110] Embodiment 2: Take the mixed test of traffic service and intelligent sensing equipment as an example for explanation.

[0111] During the mixed test of access services and smart sensing devices, the processing link of sensing service data is shorter, and the server processing speed is much higher than that of access service data. As a result, when the same number of simulators and a given reporting time interval are set, the sensing service data can reach the set load threshold in a short period of time and occupy a higher level of server processing resources, causing the access service processing speed to slow down in a manner inconsistent with the actual situation. This is different from the actual business model (the real-time requirements for access time processing are high, and the sensing data processing has asynchronous tolerance). Therefore, the data reporting time interval of the smart sensing device will be appropriately adjusted in the dynamic evaluation plan to better match the actual business model and obtain a more reasonable performance evaluation result.

[0112] The dynamic performance evaluation and testing method for the hybrid access scenario of IoT devices provided in this application adds a mechanism for real-time monitoring and dynamic linkage adjustment for different IoT services in the process of mixed IoT service performance evaluation. It mainly adjusts the business thresholds and dynamic durations of IoT services corresponding to different businesses set according to the business growth model obtained according to actual business data through the linkage of the monitoring module, control module and simulator during the stress test, and performs controllable dynamic load evaluation on IoT services / backends. Compared with the current fixed business volume or step-shaped growth load simulation method, the solution provided in this application can timely adjust the business data reported by the simulation device according to the expected business change model corresponding to different IoT devices, and is more targeted and predictable in the risk assessment of the carrying capacity of the IoT platform under business fluctuations. The method provided in this application has a wide range of applications and high utilization value.

[0113] The protection scope of the dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices described in the embodiment of the present application is not limited to the execution order of the steps listed in the present embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0114] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the following is achieved: Figure 1 The dynamic performance evaluation and testing method of the hybrid access scenario of the Internet of Things devices.

[0115] At any possible level of technical detail combination, the present application may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0116] Computer readable storage media can be tangible devices that can hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the above.

[0117] An embodiment of the present application also provides a dynamic performance evaluation and testing system for hybrid access scenarios of Internet of Things devices. The dynamic performance evaluation and testing system for hybrid access scenarios of Internet of Things devices can implement the dynamic performance evaluation and testing method for hybrid access scenarios of Internet of Things devices described in the present application. However, the implementation device of the dynamic performance evaluation and testing method for hybrid access scenarios of Internet of Things devices described in the present application includes but is not limited to the structure of the dynamic performance evaluation and testing system for hybrid access scenarios of Internet of Things devices listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0118] The following will describe in detail the dynamic performance evaluation and testing system for hybrid access scenarios of IoT devices provided by this embodiment with reference to the diagrams.

[0119] This embodiment provides a dynamic performance evaluation and testing system for a hybrid access scenario of IoT devices, including:

[0120] See also Fig.11 , which is a schematic diagram of the principle structure of a dynamic performance evaluation and testing system for hybrid access scenarios of IoT devices described in this application in one embodiment. Fig.11 As shown, the dynamic performance evaluation and testing system for the hybrid access scenario of the Internet of Things devices includes: a simulator setting module 111, a data generation module 112, a data storage and monitoring module 113, a data comparison module 114, and a business control module 115.

[0121] The simulator setting module 111 is used to set device simulators for various IoT services and configure key service parameters.

[0122] In this embodiment, corresponding device simulators can be set according to various business system devices and the protocols used by the business system devices to obtain device simulators corresponding to each business system; the device simulators include but are not limited to: a traffic device simulator, a sensing device simulator, a smart screen device simulator, etc.; business parameters of each device simulator are configured according to business requirements. The key business parameters include but are not limited to: device attributes, reporting cycle, reporting frequency, reporting data details, etc.

[0123] The data generating module 112 is configured to generate business data based on the key business parameters.

[0124] In this embodiment, data reporting rules are formulated according to business requirements; data generation logic is designed based on the key business parameters and the data reporting rules; and simulated business data is generated according to the data generation logic.

[0125] In this embodiment, first, the communication parameters required for business data reporting (such as IP address, port number, communication protocol, etc.) are configured in the business system device simulator (such as a traffic device simulator, a sensing device simulator, a smart screen device simulator, etc.), and each simulator is started to establish a communication connection relationship with the business data reporting.

[0126] Furthermore, according to the business data reporting requirements, the format of the reporting data of the business system device simulator is configured. Then, the device operation status is simulated in each device simulator, and the corresponding business data is generated. The business data is then converted according to the configured data format and reported to the business data control module through the communication connection.

[0127] The data storage and monitoring module 113 is used to obtain business data flow information based on the business data storage status.

[0128] In this embodiment, business data storage information within a certain period of time is obtained; and business data flow information is obtained based on the actual business data storage information.

[0129] The data comparison module 114 is used to compare the service data flow information with the service data control threshold interval to generate an adjustment request.

[0130] In this embodiment, each business data control threshold is set according to business needs; the business data flow information is compared with the business data control threshold to determine whether the business data reaches the trigger condition; when the business data flow information reaches the trigger condition, an adjustment request is generated.

[0131] Specifically, by reading the storage status of corresponding business data in the database in real time, combined with the control thresholds of various businesses that have been set, the adjustment request is reported to the business control module in a timely manner.

[0132] When the business data flow information exceeds the upper and lower limits of the business data control threshold, it is judged to be an abnormal state, then the adjustment mechanism is triggered and an adjustment request is generated; when the business data flow information is between the upper and lower limits of the business data control threshold, it is judged to be a normal state, then the adjustment mechanism is not triggered.

[0133] The service control module 115 is used to adjust the configuration of the corresponding type of device simulator according to the adjustment request, and update the configured service parameters.

[0134] In this embodiment, the configuration amount that needs to be adjusted is calculated based on the adjustment request, the business data control threshold and the expected limit of business growth; the configuration of the business system device simulator is adjusted according to the adjustment amount; wherein the adjustment content includes but is not limited to: data volume, frequency, duration, etc.; after completing the adjustment of the configuration of the device simulator, the device simulator is restarted and the effectiveness verification is performed; the updated business parameter configuration is adopted, and the new business data is reported.

[0135] From the above, we can see that by adding a business control module, based on the adjustment request reported by the monitoring module, combined with the configuration threshold and the expected upper and lower limits of business growth, the simulator configuration of the relevant equipment can be automatically adjusted, and the device simulator can be restarted after the adjustment is completed to ensure that the device simulator uses the new business parameter configuration to report business data.

[0136] The dynamic performance evaluation and testing system for hybrid access scenarios of IoT devices can timely adjust the business data reported by simulated devices according to the expected business change models corresponding to different IoT devices, and is more targeted and predictable in risk assessment of the carrying capacity of the IoT platform under business fluctuations.

[0137] It should be noted that it should be understood that the division of the various modules of the above system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, the above steps or modules can be completed by hardware integrated logic circuits in the processor element or instructions in software form.

[0138] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0139] See also Fig.12 , which is a schematic diagram of the principle structure of an electronic device for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices of the present application in one embodiment. Fig.12 As shown, this embodiment provides an electronic device for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices, and the electronic device for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices includes: a processor 121 and a memory 122; the memory 122 is used to store computer programs; the processor 121 is connected to the memory 122, and is used to execute the computer program stored in the memory 122, so that the electronic device for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices performs each step of the dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices as described above.

[0140] Preferably, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0141] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0142] In summary, the present application provides a method and system for dynamic performance evaluation and testing of hybrid IoT device access scenarios, which has the following beneficial effects:

[0143] The dynamic performance evaluation and testing method for the hybrid access scenario of IoT devices provided in this application adds a mechanism for real-time monitoring and dynamic linkage adjustment for different IoT services in the process of mixed IoT service performance evaluation. It mainly adjusts the business thresholds and dynamic durations of IoT services corresponding to different businesses set according to the business growth model obtained according to actual business data through the linkage of the monitoring module, control module and simulator during the stress test, and performs controllable dynamic load evaluation on IoT services / backends. Compared with the current fixed business volume or step-shaped growth load simulation method, the solution provided in this application can timely adjust the business data reported by the simulation device according to the expected business change model corresponding to different IoT devices, and is more targeted and predictable in the risk assessment of the carrying capacity of the IoT platform under business fluctuations. The method provided in this application has a wide range of applications and high utilization value.

[0144] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A dynamic performance evaluation and testing method for hybrid access scenarios of IoT devices, characterized in that: Applied to multiple IoT systems, the method includes: Set up device simulators for various IoT services and configure key service parameters; Generate and report business data based on the key business parameters; Obtain business data flow information based on business data storage conditions; Comparing the service data flow information with the service data control threshold interval to generate an adjustment request; According to the adjustment request, the configuration of the device simulator of the corresponding type is adjusted, and the configured service parameters are updated.

2. The method for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices according to claim 1 is characterized in that: Set up device simulators for various IoT services and configure key service parameters including: According to various business system devices and the protocols used by the business system devices, corresponding device simulators are set; the device simulators include: a pass device simulator, a sensing device simulator, and a smart screen device simulator; According to business requirements, business parameters are configured for the device simulator; wherein the key business parameters include: device attributes, reporting cycle, reporting frequency, and reporting business data / format.

3. The method for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices according to claim 1 is characterized in that: The business data includes: traffic data, sensing data, and smart screen status data; the business data generated and reported based on the key business parameters includes: Formulate data reporting rules based on business needs; Designing data generation logic based on the key business parameters and the data reporting rules; The simulated business data is generated according to the data generation logic.

4. The method for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices according to claim 1, characterized in that: Comparing the service data flow information with the service data control threshold interval and generating an adjustment request includes: Set control thresholds for each business data based on business needs; Compare the service data flow information with the service data control threshold to determine whether the service data flow information reaches a trigger condition; When the business data reaches the trigger condition, an adjustment request is generated.

5. The method for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices according to claim 4 is characterized in that: Comparing the service data flow information with the service data control threshold to determine whether the service data reaches the trigger condition includes: When the service data flow information exceeds the upper and lower limits of the service data control threshold, it is judged as an abnormal state, an alarm mechanism is triggered, and an adjustment request is generated; When the service data flow information is within the upper and lower limit intervals of the service data control threshold, it is judged to be in a normal state and the alarm mechanism is not triggered.

6. The method for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices according to claim 1, characterized in that: Also includes: Before the service data is reported in batches, the service control module is started, and the service data of the corresponding type is reported according to the initial service configuration parameters or the service configuration parameters adjusted after the adjustment request is triggered.

7. The method for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices according to claim 1, characterized in that: According to the adjustment request, adjusting the configuration of the corresponding type of device simulator includes: Calculating the configuration amount to be adjusted according to the adjustment request, the service data control threshold and the expected service growth limit; The configuration of the business system device simulator is adjusted according to the adjustment amount; wherein the adjustment content includes: data volume, frequency, and duration.

8. The method for dynamic performance evaluation and testing of hybrid access scenarios of IoT devices according to claim 1, characterized in that: Also includes: After completing the adjustment of the configuration of the device simulator, restart the device simulator and perform a validation; The updated service parameter configuration is adopted to report the new service data.

9. A dynamic performance evaluation and testing system for hybrid access scenarios of IoT devices, characterized in that: include: The simulator setting module is used to set up device simulators for various IoT services and configure key service parameters; A data generation module, used to generate and report business data based on the key business parameters; Data storage monitoring module, used to obtain business data flow information based on business data storage conditions; A data comparison module, used to compare the service data flow information with the service data control threshold interval and generate an adjustment request; The service control module is used to adjust the configuration of the device simulator of the corresponding type according to the adjustment request, and update the configured service parameters.

10. An electronic terminal, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory, so that the dynamic performance evaluation and testing device for the hybrid access scenario of the Internet of Things device executes the dynamic performance evaluation and testing method for the hybrid access scenario of the Internet of Things device as described in any one of claims 1 to 8.

Images