Method, apparatus, device, medium and product for driving tests based on existing network data

Abstract

An embodiment of the present application provides a method, device, equipment, medium and product based on in-network data-driven testing, relating to the technical field of 5G networks. The method includes: grouping the collected users according to the quality of service levels defined in the 3GPP protocol to obtain at least one user group; creating a user queue for each user group according to the start time of the corresponding historical service of each user, and sequentially obtaining test cases of each user in the corresponding user group according to each user queue, and running each test case in a test network environment; the test case of each user is obtained according to the corresponding historical service and historical session of the user; finally, analyzing the operation data collected from the test network environment to obtain a test result. The solution provided by the embodiment of the present application can narrow the gap between the test result in the laboratory network environment and the usage effect in the real network environment.

Description

Technical Field

[0001] This application relates to the technical field of 5G network, and more particularly, to a method, device, electronic device, computer-readable storage medium, and computer program product for testing based on live network data-driven testing. Background Art

[0002] Currently, for the communication simulation testing of the 5G core network, test engineers manually write test cases based on the analysis and understanding of the 3GPP protocol, and then conduct tests through simulation devices. The test cases generated in this way usually execute services with a fixed number of users and a fixed traffic volume, belonging to a static test model. However, the services of live network users are often dynamic, and the entire live network service model is very complex. If the network is simply tested with a static model, there may be a large difference between the test results in the laboratory network and the actual usage effects in the live network. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide a method, device, electronic device, computer-readable storage medium, and computer program product for testing based on live network data-driven testing, so as to solve the above technical problems. To achieve this purpose, the embodiments of this application provide the following several solutions.

[0004] On the one hand, the embodiments of this application provide a method for testing based on live network data-driven testing, and the method includes:

[0005] Group the collected users according to a preset quality of service level to obtain at least one user group; the information of each user includes historical services initiated in the production network environment and historical sessions associated with the historical services, and the historical sessions include configured quality of service; create a user queue for each user group according to the start time of the corresponding historical service of each user, and sequentially obtain test cases for each user in the corresponding user group according to each user queue, and run each test case in the test network environment; the test case for each user is obtained according to the corresponding historical service and historical session of the user; analyze based on the operation data collected from the test network environment.

[0006] Optionally, grouping the collected users according to a preset quality of service level to obtain at least one user group includes:

[0007] Collect historical data of all users in a preset scenario in the production network environment; the historical data includes historical sessions and historical services; group all users according to the quality of service in the historical session of each user to obtain at least one user group; wherein, the quality of service corresponding to each user in each user group corresponds to the same quality of service level.

[0008] Optionally, before obtaining the test cases of each user in the corresponding user group according to each user queue, the method includes:

[0009] Obtain the test scripts corresponding to each user group; the test scripts include the quality of service corresponding to the quality of service level to which the user group belongs; generate test template cases based on the test scripts corresponding to each user group.

[0010] Optionally, each historical service includes the service type, start time, and duration of the historical service; obtaining the test cases of each user in the corresponding user group according to each user queue includes:

[0011] According to the order of the users in the user queue, input the registration information of each user, the service type, start time, and duration of the corresponding historical service into the test template corresponding to the user group, and obtain the test cases corresponding to each user in the user group.

[0012] Optionally, running each test case in the test network environment includes:

[0013] Layout a test session in the test network environment based on the test case; the test session is consistent with the quality of service of the corresponding historical session; layout a test service on the test session based on the test case; the test service is consistent with the service type, start time, and duration of the corresponding historical service.

[0014] Optionally, perform performance analysis based on the operation data collected from the test network environment, including performing the following operations for each user group:

[0015] Obtain the operation data cached by multiple network elements in the test network environment according to at least one preset metric; analyze the operation performance of the core network in the test network environment according to the operation data.

[0016] Optionally, analyzing the operation performance of the core network in the test network environment according to the operation data includes:

[0017] If the operation data meets the preset threshold, determine to end the simulation test; if the operation data does not meet the preset threshold, adjust the configuration parameters of the core network in the test network environment, and determine to start the next round of simulation test. On the other hand, an embodiment of the present application provides a dynamic test device, and the device includes:

[0018] A grouping module, configured to group the collected users according to a preset quality of service level to obtain at least one user group; the information of each user includes the historical services initiated in the production network environment and the historical sessions associated with the historical services, and the historical sessions include the configured quality of service.

[0019] A simulation test module, configured to create a user queue for each user group according to the start time of the historical service corresponding to each user, and sequentially obtain test cases of each user in the corresponding user group according to each user queue, and run each test case in a test network environment; the test case of each user is obtained according to the historical service and historical session corresponding to the user.

[0020] An analysis module, configured to perform analysis based on the operation data collected from the test network environment.

[0021] An embodiment of the present application further provides an electronic device, which includes: a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of a method for data-driven testing based on live network data provided by an embodiment of the present application.

[0022] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of a method for data-driven testing based on live network data provided by an embodiment of the present application are implemented.

[0023] An embodiment of the present application further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of a method for data-driven testing based on live network data provided by an embodiment of the present application are implemented.

[0024] The beneficial effects brought by the technical solution provided by the embodiment of the present application are:

[0025] The embodiment of the present application provides a method for driving tests based on live network data. By collecting real user data from the production network environment, such as the sessions and service data of real users, it realizes simulating the behaviors of live network users from the perspective of real data. By grouping the collected users according to the corresponding quality of service levels, generating test cases corresponding to each user (i.e., generating a dynamic test model), and running the test cases, the test effects corresponding to different qualities of service can be obtained. Based on the test cases, simulation tests can be quickly realized, thereby improving the simulation efficiency. That is to say, the solution provided by the embodiment of the present application generates a dynamic test model (such as a test case) by capturing various data generated in the 5G core network of the production network environment (such as the service data and session data of real users), and drives the operation of the test network through this dynamic test model. Compared with the simulation tests based on a fixed number of users and fixed services in the prior art, the simulation test solution provided by the embodiment of the present application is based on real user data, and this user data can also be collected from the production network environment at any time according to requirements. Since the data source comes from the real network, it ensures that the business implementation process in the test network environment is closer to the business implementation process in the production network environment, effectively narrowing the gap between the test results of the core network in the test network environment and the usage effects of the core network in the production network environment. Description of the Drawings

[0026] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments of the present application.

[0027] Figure 1a A schematic diagram of the test architecture of a 5G core network provided by the related art;

[0028] Figure 1b A schematic diagram of the position of NWDAF in the 5G core network provided by the related art;

[0029] Figure 2 A schematic flowchart of a method for driving tests based on live network data provided by the embodiment of the present application;

[0030] Figure 3 A schematic diagram of the structure of the test networking model provided by the embodiment of the present application;

[0031] Figure 4 A schematic diagram of the structure of a device for driving tests based on live network data provided by the embodiment of the present application;

[0032] Figure 5 A schematic diagram of the structure of an electronic device provided by the embodiment of the present application. Detailed Embodiments

[0033] The embodiments of the present application will be described below in conjunction with the accompanying drawings in the present application. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0034] Those skilled in the art of the present technology can understand that unless specifically stated, the singular forms "a", "an", "the", and "said" used herein may also include the plural forms. It should be further understood that the terms "include" and "comprise" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude the implementation of other features, information, data, steps, operations, elements, components, and / or their combinations supported by the art of the present technology. It should be understood that when we say an element is "connected" or "coupled" to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or implemented as "B", or implemented as "A and B".

[0035] To make the objectives, technical solutions, and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0036] A mobile communication network generally consists of an access network, a bearer network, and a core network. Among them, the core network is the key to the stability and robustness of the entire mobile communication network. Whether the networking structure and parameter settings of the core network are appropriate can directly affect the user experience. Therefore, the simulation test for the core network has always been the focus of mobile communication network design.

[0037] Currently, the testing process of mobile communication networks generally involves simulating certain network elements to test other network elements. For example, in the case of an LTE network, UE / eNB (User Equipment / evolved Node B) can be simulated to conduct end-to-end testing of the LTE network; or a certain node in the core network of the LTE network can be simulated. For instance, MME (Mobility Management Entity) can be simulated to test SGW (a type of network element), and SGW (Serving GW) can be simulated to test PGW (PDN Gateway, Packet Data Network Gateway). In the case of a 5G network, UE / gNB (User Equipment / gNode B) can be simulated to conduct end-to-end testing of the 5G network; or AMF (Access and Mobility Management Function) can be simulated to test SMF (Session Management Function), and SMF can be simulated to test UPF (User Plane Function), etc. When conducting simulation testing in the above manner, the process usually involves specifying the test model, the number of users, and the user service type, and then conducting tests with specified traffic. Especially for performance and stability testing, the experimental environment is configured according to the maximization of the live network layout. Obviously, the above testing process belongs to a static testing.

[0038] Figure 1a Fig. shows a testing architecture of a 5G core network. In this architecture, UE / gNB is simulated to achieve the performance and stability testing of the 5G core network. In the 3GPP protocol, artificial intelligence is introduced. For example, a new network element, NWDAF (Network Data Analytics Function), is added to the core network. NWDAF is a data-aware analysis network element that is based on network data and participates in the entire life cycle of network planning, construction, operation and maintenance, network optimization, and operation. Specifically, NWDAF collects information such as user connection management, mobility management, session management, and accessed services, uses reliable analysis and prediction models to evaluate and analyze different types of users, constructs user portraits, determines the user's movement trajectory and service usage habits, and predicts user behavior. Based on the analysis and prediction data, user mobility management parameters and radio resource management parameters are optimized.

[0039] Figure 1b Fig. shows a schematic diagram of the location of NWDAF in the 5G network. For example, some network elements in the 5G core network are connected to NWDAF.

[0040] Continuing from the background art, under the condition of having NWDAF, the communication simulation testing for the 5G core network is still that the test engineer manually writes test cases based on the analysis and understanding of the 3GPP protocol and then conducts testing through simulation devices. Therefore, this simulation testing is still a static testing. If this simulation testing is executed, it will result in a large difference between the laboratory network test results and the actual usage effects in the live network.

[0041] To solve the above technical problems, an embodiment of the present application provides a method based on in-network data-driven testing, which is applied to any terminal device, such as a server. Specifically, the method collects user data (such as historical services, historical sessions associated with historical services, and historical sessions including quality of service (QoS)) from a real 5G core network, groups users according to corresponding QoS levels to obtain multiple user groups, creates corresponding test cases based on the historical services and historical sessions of each user in each user group, and runs each test case in a test network environment. Finally, the running data collected from the test network environment is analyzed to obtain test results. Compared with the prior art, the solution provided by the embodiment of the present application can ensure that the service implementation process in the test network environment is closer to the service implementation process in the production network environment, effectively narrowing the gap between the test results of the core network in the test network environment and the usage effects of the core network in the production network environment.

[0042] Optionally, the method provided by the embodiment of the present application can be implemented as an independent application program or a functional module / plugin of an application program. For example, the application program can be a dedicated in-network data-driven testing application or other application programs with in-network data-driven testing functions. Through this application program, the effect of narrowing the gap between the test results in the laboratory network environment and the usage effects in the real network environment can be achieved.

[0043] To better illustrate the solution provided by the embodiment of the present application, the technical solution of the embodiment of the present application and the technical effects generated by the technical solution of the present application will be described below through the description of several exemplary embodiments. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. For the same terms, similar features, and similar implementation steps in different embodiments, they will not be described repeatedly.

[0044] Figure 2 FIG. shows a schematic flowchart of a method based on in-network data-driven testing. As Figure 2 shown, the method includes steps S210 to S230.

[0045] S210, group the collected users according to a preset quality of service level to obtain at least one user group; the information of each user includes historical services initiated in the network environment, and historical sessions associated with the historical services, and the historical sessions include configured quality of service.

[0046] Among them, the production network environment is the operating environment of the core network in the real 5G network. Users can initiate service requests in this operating environment and enjoy network services. The test network environment is the operating environment of the core network in the experimental 5G network. Among them, the core networks in the production network environment and the test network environment are both configured with NWDAF.

[0047] Among them, the user's information also includes the user's registration information. Among them, the registration information at least includes the user's unique identifier, such as the telephone number assigned to the user by the network operator.

[0048] Optionally, S210 may specifically include: collecting data through the NWDAF in the core network of the production network environment, and the data includes real user data generated in the production network environment.

[0049] Among them, when collecting data, it can be carried out according to a preset scenario. For example, collecting data for all audiences who initiate services in a certain concert scenario.

[0050] In a possible implementation manner of this embodiment, grouping the collected users according to a preset quality of service level to obtain at least one user group, which may specifically include:

[0051] Collecting all users in the preset scenario in the production network environment; grouping all users according to the corresponding quality of service of each user to obtain at least one user group; among them, the corresponding quality of service of all users in each user group corresponds to the same quality of service level.

[0052] Among them, the corresponding quality of service of each user is the quality of service in the historical session associated with the historical service initiated by the user, that is, QoS.

[0053] Among them, the preset quality of service level is the multiple qualities of service defined in the 3GPP protocol. Specifically, the 3GPP protocol stipulates that for each initiated session, one quality of service needs to be obtained from multiple qualities of service and configured for the session, so as to provide network services for the services on the session according to the standard of this quality of service. Therefore, when grouping, the preset quality of service level is the quality of service level defined by the 3GPP protocol. The multiple quality of service levels stipulated by the 3GPP protocol are shown in Table 1 below.

[0054] Table 1

[0055]

[0056] After classifying the collected users according to the quality of service levels shown in Table 1, at most n user groups can be obtained. Among them, each quality of service level can be identified by 5QI. For example, user group 1 can be identified by 5QI = 1.

[0057] Among them, each historical service includes the service type, start time, and duration of the historical service. The end time of the historical service is obtained according to the duration and start time. Next, the start times and durations of the historical services of each user in each user group are classified to obtain the data shown in Table 2.

[0058] Table 2

[0059]

[0060]

[0061] Next, taking user group 1, start time 1, and end time 1 as examples, the data in Table 2 will be described. 50 users in user group 1 initiate a signaling plane process at 14:00 and start a media plane process with a certain 5QI service quality level, and the service lasts until 15:00. Other users can also initiate services between 14:00 and 15:00. The data in Table 2 is collected by NWDAF from the production network environment.

[0062] S220, create a user queue for each user group according to the start time of the corresponding historical service of each user, and sequentially obtain the test cases of each user in the corresponding user group according to each user queue, and run each test case in the test network environment; the test case of each user is obtained according to the corresponding historical service and historical session of the user.

[0063] In one implementation manner of this embodiment, since the service types corresponding to the service quality levels are different, and the service types corresponding to one service quality level are basically the same. Therefore, corresponding test scripts can be created and stored in advance according to each service quality level.

[0064] Among them, the stored test script can be understood as a test template. In the process of generating the test cases corresponding to each user, due to the different registration information of the input users and the time information of the services, the cases finally generated by the same test script are different. Therefore, corresponding test template cases can be generated for each user group. Then, fill in the differential data in the final stage of generating test cases to generate the test cases corresponding to each user.

[0065] In an example, the test script can be stored in the json format. Since the json format is a typical format of "key" and "value" in pairs, the "value" in the json script can be in the default state. For example, for the same user group, a test script is used, and the registration information, time information, etc. of the users in the test script can be in the default state.

[0066] Optionally, before obtaining the test cases of each user in the corresponding user group according to each user queue, the method further includes:

[0067] Obtaining a test script corresponding to the quality of service level of each user group; generating test template cases based on the test script corresponding to each user group.

[0068] Wherein, the test script includes the quality of service corresponding to the quality of service level to which the user group belongs; the test script further includes the following processes: a signaling plane process related to the session, and a media plane process related to the service.

[0069] Exemplarily, the following is a json test script corresponding to the QoS with 5QI being 1. The part of the content related to the quality of service in this script is as follows:

[0070] ……

[0071] QoS:{

[0072] 5QI:1;

[0073] Resource Type:GMBR; / / Resource type

[0074] Default Priority Level:20; / / Priority level

[0075] Packet Delay Budget:100ms; / / Packet delay budget

[0076] Packet Error:0.01; / / Packet error rate

[0077] Default Maximum Data Burst Volume:N / A; / / Maximum data burst volume

[0078] Averaging Window:2000ms; / / Aggregate bit rate

[0079] }……

[0080] After generating the test template cases, the registration information of the users in the test template cases, the start time and duration of the historical services are all in the default state. The test template cases of the user groups created above belong to the test cases for a certain type of QoS service. Since there are differences in the registration information and historical sessions (i.e., session tunnel information) of each user in each user group, test template cases corresponding to each user group can be generated, and the registration information and session tunnel information of the users are set as default parameters. In actual operation, the registration information and session tunnel information of a certain user are replaced into the test template cases to generate the corresponding test cases for that user.

[0081] In one implementation manner of this embodiment, each historical service includes the service type, start time and duration of the historical service; obtaining the test cases of each user in the corresponding user group according to each user queue in turn may specifically include:

[0082] Perform the following operations on the user group corresponding to each user queue:

[0083] Input the registration information of each user, the service type, start time and duration of the corresponding historical service into the test template corresponding to the user group in the order of the users in the user queue to obtain the test cases corresponding to each user in the user group.

[0084] Among them, after obtaining the test cases corresponding to each user, run the test cases.

[0085] Among them, there is a distributed execution engine in the test network environment. When the user group is one or more, the distributed engine can be called to execute. For example, each distributed execution engine is responsible for the execution process of all test cases in a user group. Among them, the distributed execution engine can simulate the UE (user terminal) or gNB (base station) and send various messages to the core network in the test network environment.

[0086] Optionally, creating a user queue for each user group according to the start time of the historical service corresponding to each user may specifically include:

[0087] Create a user queue based on the start time of the historical service corresponding to each user; the user queue represents the execution order of the respective simulation tests of all users in the group. Therefore, according to the execution order represented by the user queue, the test template cases are filled in the above manner for each user in turn, so as to obtain the test cases corresponding to each user and perform simulation tests through the test cases.

[0088] Specifically, a user queue is created in each user group based on the start time and duration of each user's corresponding historical service. When it comes to a specific user, the user's registration information and historical session (i.e., session tunnel information) are replaced into the test template case to form a test case belonging to this user. Running this test case can complete the simulation of the user's historical service. Among them, due to the simultaneous operation of user services in multiple user groups, a distributed execution engine can be used for simulation testing. For example, each execution engine runs the test cases corresponding to each user in a user group.

[0089] Among them, the simulation tests corresponding to users in the execution queue of each user group can be cross-executed with each other. For example, when executing the media plane process for User 1 in User Group 1, the signaling plane process can also be executed for User 2 in User Group 2 at the same time.

[0090] To understand this process of creating a queue more clearly, the embodiment of this application also provides a pseudo-code example for detailed description. This example provides two data structures, the data structure of the execution queue, and the data structure of the information to be filled in related to each user.

[0091] Typedef UserGroupExeInfo / / Execution queue

[0092] {

[0093] int UserGroupID; / / Unique identifier of the user group;

[0094] Queue <subusergroupinfo>SubUserGrougQueue; / / Execution queue;

[0095] }

[0096] Typedef SubUserGroupInfo / / Information to be filled in for each user

[0097] {

[0098] Time StartExecuteTime; / / Start time of the business;

[0099] Int UserID[MAX_USER_NUM]; / / User's unique ID;

[0100] Time LastTime; / / Duration of the business;

[0101] }

[0102] In an implementation of this embodiment, running each test case in the test network environment may specifically include the following steps Sa1 to Sa2.

[0103] Sa1, layout of test sessions in the test network environment based on test cases; the quality of service of the test sessions is consistent with that of the corresponding historical sessions;

[0104] Specifically, the execution engine creates a simulation instance about the UE or gNB according to the instruction of the test case, and initiates a session creation request to the core network in the test network environment through the simulation instance to request the creation of a test session. The session creation request also carries the quality of service so that the core network in the test network environment configures the quality of service for the test session.

[0105] Sa2, layout of test services on the test session based on the test case; the test services are consistent with the service type, start time and duration of the corresponding historical services.

[0106] Specifically, the execution engine calls the simulation instance according to the instruction of the test case to initiate a service request to the core network in the test network environment through the test session to request a network service.

[0107] In one implementation of this embodiment, when running each test case, it also includes:

[0108] An account opening request is initiated to the UDM (unified data management network element) in the test network environment according to the user's registration information. The UDM receives the account opening request and after performing the account opening operation, can lay out a test session and start a test service.

[0109] S230, perform performance analysis based on the operation data collected from the test network environment.

[0110] In one implementation of this embodiment, when collecting operation data, the following operations can be specifically performed for each user group:

[0111] Obtain the operation data cached by multiple network elements in the test network environment according to at least one preset index; analyze the operation performance of the core network in the test network environment based on the operation data.

[0112] Specifically, the NWDAF network element can be used to obtain the operation data cached by multiple network elements.

[0113] Optionally, the operation data can include access success rate, service rate, and user delay.

[0114] Among them, the setting of the target network element can also be judged according to the collected operation data.

[0115] Specifically, call the NWDAF in the test network environment to obtain operation data from each connected network element. For example, data collection can be performed according to the specific transactions responsible for each network element. Optionally, collect access control-related data from the AMF network element, such as how many session creation requests are accessed and whether they are all created successfully; optionally, collect verification data from the UPF network element, such as whether the data packet is forwarded to the corresponding terminal; optionally, collect uplink and downlink rates from the UPF network element. It should be noted that data can also be collected from other network elements of the 5G core network, which will not be elaborated here for simplicity of description.

[0116] In one implementation of this embodiment, analyzing the operation performance of the core network in the test network environment based on the operation data can specifically include:

[0117] If the operation data meets the preset threshold, determine to end the simulation test.

[0118] If the operation data does not meet the preset threshold, adjust the configuration of the core network in the test network environment and determine to start the next round of simulation test.

[0119] Optionally, the preset threshold can be one or more. For example, set a threshold for the access success rate of the session to check whether the quantity planning of the AMF and / or SMF is appropriate; for example, create a threshold for the uplink and downlink rates to check whether the quantity planning of the UPF network element is appropriate. It should be noted that other thresholds can also be set for corresponding checks.

[0120] Among them, after determining that the operation data meets the threshold, various configurations of the core network in the test network environment can be adjusted, such as the configured number of network elements, such as the parameter configuration of network elements. Specifically, when the access success rate is less than the corresponding threshold, the AMF network element or the SMF network element is added; when the uplink and downlink rates are less than the corresponding threshold, one or more UPF network elements are added, or the settings of the UPF are changed.

[0121] A method based on live network data-driven testing provided by an embodiment of the present application can be applied to various test terminals (such as servers), and a request is sent to the test network environment through the test terminal to test whether the configuration of the core network meets the requirements. In order to more clearly describe the technical effects of the method, an embodiment of the present application provides a specific example based on a test terminal for illustration. Among them, the example includes a test networking model and a test process.

[0122] See Figure 3 In the shown test networking model, there are two operating environments of the core network, namely the production network environment and the test network environment. Based on Figure 3 In the shown test networking model, this example also provides a test process, which includes steps S1001 to S1005.

[0123] S1001, the terminal collects user data from the production network environment.

[0124] The terminal collects user data from the NWDAF in the production network environment, such as user registration information, historical services initiated by the user, and historical sessions associated with each historical service. Among them, each historical session includes the configured QoS (Quality of Service), and each historical service specifically includes the start time and duration of the service.

[0125] S1002, the terminal groups the collected user data.

[0126] The terminal calls the classification model shown in Table 1 in the above embodiment to classify users and generate N user groups. Among them, all users in each user group have the same corresponding QoS.

[0127] S1003, the terminal creates test cases for each user group.

[0128] The terminal calls the json script corresponding to each user group to generate a test template case. Since the user registration information in each user group is different, the test template case can be generated first. When the specific queue executes to a certain user, the registration information of the user is replaced into the test template case to form a test case belonging to the user.

[0129] The terminal sends N user groups and the corresponding initial test template cases to the distributed execution engine. Each execution engine is responsible for at least the execution process of the test cases of one user group.

[0130] Among them, the terminal can also generate a service control queue for each user group. For example, for any user in a certain user group, if the service initiation time of a user is earlier, the user will be at the front of the service control queue, and the corresponding test case of the user will be executed earlier.

[0131] S1004, the execution engine executes the test case.

[0132] For one of the execution engines: when obtaining a test case, first fill the initial test case with the information of a certain user (such as the user's registration information, session tunnel information, service information) to obtain the corresponding test case of the user. Among them, during the process of running the test case of each user, an account opening operation needs to be initiated according to the user's registration information, and then a session tunnel is created according to the user's session tunnel information, and the service is executed through this session tunnel.

[0133] In addition, a timer is also configured in the execution engine. As shown in Table 2 of the above embodiment, since there are 2 batches of users with different start times and durations in the same user group, the execution duration can be set for each batch of users. Once the execution duration is reached, the operation of the test session channel is stopped.

[0134] Among them, when the user uses the services of the core network, it involves the layout of two processes, one is the signaling plane process and the other is the media plane process. Running the corresponding test case of each user is equivalent to laying out the signaling plane process in the core network. In step S1001, in addition to collecting the user data of each user, the service data of each user can also be collected so as to execute the media plane process according to the service data.

[0135] S1005, the terminal evaluates the networking model in the test network environment.

[0136] Specifically, the NWDAF in the test network environment collects operation data from multiple network elements. For example, the uplink and downlink rates are collected from the UPF network element.

[0137] The terminal determines whether the collected operation parameters meet the expected thresholds. If the expected thresholds are not reached, the core network in the test network environment is adjusted, and the user data is obtained again from the production network environment so as to start the test again.

[0138] This example also makes the following comparison between the dynamic test shown in this test process and the existing static test:

[0139]

[0140]

[0141] Based on the above test process, it can be seen that the test process shown in this example can more realistically simulate the user behavior in a real network. By combining to generate dynamic test cases, and based on the dynamic test cases, it is verified that the test results of the laboratory network are more accurate than ordinary static test methods and are closer to the usage effect in a real network. Therefore, the test results can help developers discover various problems existing in the real network as early as possible, and timely propose modification plans to avoid the problems being exposed to the real network environment.

[0142] Figure 4 A device 400 based on live network data-driven testing is shown. Among them, the device 400 includes the following modules:

[0143] A grouping module, configured to group the collected users according to a preset quality of service level to obtain at least one user group; the information of each user includes historical services initiated in the production network environment and historical sessions associated with the historical services, and the historical sessions include configured quality of service;

[0144] A simulation test module, configured to create user queues for each user group according to the start time of the corresponding historical service of each user, and sequentially obtain test cases of each user in the corresponding user group according to each user queue, and run each test case in the test network environment; the test case of each user is obtained according to the corresponding historical service and historical session of the user;

[0145] An analysis module, configured to analyze based on the operation data collected from the test network environment.

[0146] Optionally, in the grouping module 410, when grouping the collected users according to a preset quality of service level to obtain at least one user group, it is specifically configured to:

[0147] Collect historical data of all users in a preset scenario in the production network environment; the historical data includes historical sessions and historical services; group all users according to the quality of service in the historical session of each user to obtain at least one user group; wherein, the quality of service corresponding to each user in each user group corresponds to the same quality of service level.

[0148] Optionally, before the simulation test module 420 sequentially obtains the test cases of each user in the corresponding user group according to each user queue, it is specifically configured to:

[0149] Obtain a test script corresponding to each user group; the test script includes the quality of service corresponding to the quality of service level to which the user group belongs; generate a test template case based on the test script corresponding to each user group.

[0150] Optionally, each historical service includes the service type, start time, and duration of the historical service; when the simulation test module 420 sequentially obtains the test cases of each user in the corresponding user group according to each user queue, it is specifically used for:

[0151] Perform the following operations on the user group corresponding to each user queue:

[0152] According to the order of the users in the user queue, input the registration information of each user, the service type, start time, and duration of the corresponding historical service into the test template corresponding to the user group, and obtain the test cases corresponding to each user in the user group.

[0153] Optionally, when the simulation test module 420 runs each test case in the test network environment, it is specifically used for:

[0154] Layout a test session in the test network environment based on the test case; the test session is consistent with the quality of service of the corresponding historical session; layout a test service on the test session based on the test case; the test service is consistent with the service type, start time, and duration of the corresponding historical service.

[0155] Optionally, when the analysis module 430 performs performance analysis based on the operation data collected from the test network environment, it includes performing the following operations on each user group:

[0156] Obtain the operation data cached by multiple network elements in the test network environment according to at least one preset index; analyze the operation performance of the core network in the test network environment based on the operation data.

[0157] Optionally, when the analysis module 430 analyzes the operation performance of the core network in the test network environment based on the operation data, it is specifically used for:

[0158] If the operation data meets the preset threshold, determine to end the simulation test; if the operation data does not meet the preset threshold, adjust the configuration of the core network in the test network environment and determine to start the next round of simulation test.

[0159] The device according to the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed function description of each module of the device, reference can be specifically made to the description in the corresponding method shown above, and details are not described herein again.

[0160] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the above computer program to implement the steps of the method. Compared with the related art, it can achieve: narrowing the gap between the test results of the core network in the test network environment and the usage effect of the core network in the production network environment.

[0161] In an alternative embodiment, an electronic device is provided, as Figure 5 shown. Figure 5 The electronic device 5000 shown includes: a processor 5001 and a memory 5003. Among them, the processor 5001 and the memory 5003 are connected, such as connected through a bus 5002. Optionally, the electronic device 5000 may further include a transceiver 5004, and the transceiver 5004 may be used for data interaction between this electronic device and other electronic devices, such as sending and / or receiving data, etc. It should be noted that in practical applications, the transceiver 5004 is not limited to one, and the structure of the electronic device 5000 does not constitute a limitation to the embodiment of the present application.

[0162] The processor 5001 may be a CPU (Central Processing Unit, central processor), a general-purpose processor, a DSP (Digital Signal Processor, data signal processor), an ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), an FPGA (Field Programmable Gate Array, field programmable gate array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of the present application. The processor 5001 may also be a combination that implements computing functions, such as a combination including one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0163] The bus 5002 may include a path for transmitting information between the above components. The bus 5002 may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus or an EISA (Extended Industry Standard Architecture, extended industry standard architecture) bus, etc. The bus 5002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0164] The memory 5003 can be a ROM (Read Only Memory), or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory), or other types of dynamic storage devices that can store information and instructions. It can also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, which is not limited herein.

[0165] The memory 5003 is used to store the computer program for implementing the embodiments of the present application, and is controlled by the processor 5001 to execute. The processor 5001 is used to execute the computer program stored in the memory 5003 to implement the steps shown in the foregoing method embodiments.

[0166] Among them, the electronic device includes but is not limited to: a server.

[0167] The embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.

[0168] The embodiments of the present application also provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.

[0169] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than the illustrated or textually described order.

[0170] It should be understood that although the flowcharts of the embodiments of the present application indicate each operation step by arrows, the execution order of these steps is not limited to the order indicated by the arrows. Unless there is a clear description in this article, in some implementation scenarios of the embodiments of the present application, the implementation steps in each flowchart can be executed in other orders according to requirements. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage among these sub-steps or stages can also be executed at different times respectively. In the scenario where the execution times are different, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and the embodiments of the present application do not limit this.

[0171] The above are only optional implementation manners of some implementation scenarios of the present application. It should be noted that for those of ordinary skill in the art, without departing from the technical concept of the solution of the present application, adopting other similar implementation means based on the technical idea of the present application also belongs to the protection scope of the embodiments of the present application.< / subusergroupinfo>

Claims

1. A method based on in-network data-driven testing, characterized in that, The method includes: Grouping the collected users according to a preset quality of service level to obtain at least one user group; the information of each user includes historical services initiated in the production network environment and historical sessions associated with the historical services, and the historical sessions include configured quality of service. Creating a user queue for each user group according to the start time of the historical service corresponding to each user, and sequentially obtaining test cases for each user in the corresponding user group according to each user queue, and running each test case in the test network environment; the test case for each user is obtained according to the historical service and historical session corresponding to the user. Performing analysis based on the operation data collected from the test network environment.

2. The method according to claim 1, wherein The grouping the collected users according to a preset quality of service level to obtain at least one user group includes: Collecting historical data of all users in a preset scenario in the production network environment; the historical data includes historical sessions and historical services. Grouping all users according to the quality of service in the historical session of each user to obtain the at least one user group. Wherein, the quality of service corresponding to each user in each user group corresponds to the same quality of service level.

3. The method according to claim 1, wherein Before sequentially obtaining test cases for each user in the corresponding user group according to each user queue, the method includes: Obtaining a test script corresponding to each user group; the test script includes the quality of service corresponding to the quality of service level to which the user group belongs. Generating a test template case based on the test script corresponding to each user group.

4. The method according to claim 3, wherein Each historical service includes the service type, start time, and duration of the historical service. The sequentially obtaining test cases for each user in the corresponding user group according to each user queue includes: Performing the following operations for the user group corresponding to each user queue: According to the order of the users in the user queue, inputting the registration information of each user, the service type, start time, and duration of the corresponding historical service into the test template corresponding to the user group to obtain test cases corresponding to each user in the user group.

5. The method according to claim 4, wherein The running each test case in the test network environment includes: Laying out a test session in the test network environment based on the test case; the test session is consistent with the quality of service of the corresponding historical session. Laying out a test service on the test session based on the test case; the test service is consistent with the service type, start time, and duration of the corresponding historical service.

6. The method according to claim 1, characterized in that, The performing performance analysis based on the operation data collected from the test network environment includes performing the following operations for each user group: Obtaining the operation data cached by multiple network elements in the test network environment according to at least one preset index. Analyzing the operation performance of the core network in the test network environment according to the operation data.

7. The method according to claim 6, characterized in that, The analyzing the operation performance of the core network in the test network environment according to the operation data includes: If the operation data meets the preset threshold, determining to end the simulation test. If the operation data does not meet the preset threshold, adjust the configuration of the core network in the test network environment and determine to start the next round of simulation tests.

8. An apparatus for data-driven testing based on existing network data, characterized in that, The device includes: A grouping module, configured to group the collected users according to a preset quality of service level to obtain at least one user group; the information of each user includes historical services initiated in the production network environment and historical sessions associated with the historical services, and the historical sessions include configured quality of service; A simulation test module, configured to create user queues for each user group according to the start time of the corresponding historical service of each user, and sequentially obtain test cases for each user in the corresponding user group according to each user queue, and run each test case in the test network environment; the test case of each user is obtained according to the corresponding historical service and historical session of the user; An analysis module, configured to analyze based on the operation data collected from the test network environment.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1-7.

11. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1-7.

Images