User service perception evaluation method, device, equipment, medium and program product

Abstract

The present application discloses a method, apparatus, device, medium and program product for evaluating user service perception. The method includes, for each first KQI data of the target bearer network being a 5G network, calculating at least one first correlation index based on the first KQI data, the value of the first KQI data under the 4G network and the Spearman correlation coefficient formula; for each first correlation index, when the absolute value of the first correlation index is greater than or equal to a first threshold, evaluating the first KQI data according to the 5G network evaluation rules to obtain a first evaluation result; and calculating a comprehensive evaluation result of the user's service perception under the service based on the residence factor, the target evaluation result and the third evaluation result. The embodiments of the present application achieve a comprehensive evaluation of the service perception of 5G users.

Description

Technical Field

[0001] The present application belongs to the technical field of wireless network planning and optimization, and in particular relates to a user service perception evaluation method, apparatus, equipment, medium and program product. Background Art

[0002] In the existing technology, methods for evaluating user service perception are generally divided into three categories: signaling analysis system, wireless performance analysis system, and the fourth generation mobile communication technology (4G) network perception evaluation system. Among them, the signaling analysis system analyzes the signaling data of a single user and a single service, further evaluates the key quality indicator (KQI) corresponding to the service, and ultimately obtains the service perception result of a single user for a single service. The wireless performance indicator analysis system statistically analyzes the signaling data of wireless network elements, and then evaluates the key performance indicator (KPI) data to obtain the network perception result of the user under the wireless network element. The 4G network perception evaluation system organically combines the signaling analysis system with the wireless performance indicator analysis, and uses the KQI data of the signaling analysis system to evaluate user service perception.

[0003] However, because 5G networks use a completely new frame structure, modulation scheme, antenna technology, and bandwidth, signaling data for the same service differs significantly between 5G and 4G networks. Consequently, KQI and KPI data also differ from those on 4G networks. Therefore, the three aforementioned methods cannot be used to comprehensively evaluate 5G users' service perception.

[0004] However, with the development of 5G networks, there is an urgent need to conduct a comprehensive evaluation of the service perception of 5G users when they make comprehensive use of 5G and 4G networks.

[0005] Therefore, how to achieve comprehensive evaluation of 5G users' service perception has become a technical problem that urgently needs to be solved. Summary of the Invention

[0006] The embodiments of the present application provide a user service perception evaluation method, device, equipment, medium and program product, which realize comprehensive evaluation of the service perception of 5G users.

[0007] In a first aspect, an embodiment of the present application provides a method for evaluating user service perception, the method comprising:

[0008] Acquiring signaling data under at least one service for evaluating user service perception;

[0009] Determining, based on the signaling data, a target bearer network for the KQI data and a residency factor for the user terminal to reside in the 5G network;

[0010] For each first KQI data where the target bearer network is a 5G network, calculate at least one first correlation index according to the first KQI data, a value of the first KQI data in a 4G network, and a Spearman correlation coefficient formula;

[0011] For each first correlation indicator, when the absolute value of the first correlation indicator is greater than or equal to the first threshold, evaluating the first KQI data according to the 5G network evaluation rule to obtain a first evaluation result;

[0012] When the absolute value of the first correlation indicator is less than the first threshold, evaluating the first KQI data according to the 4G network evaluation rule to obtain a second evaluation result;

[0013] For each second KQI data whose target bearer network is a 4G network, the second KQI data is evaluated according to the 4G network evaluation rule to obtain a third evaluation result.

[0014] A comprehensive evaluation result of the user's service perception under the service is calculated based on the residence factor, the target evaluation result and the third evaluation result.

[0015] In some embodiments, after determining abnormal KQI data in the to-be-determined abnormal KQI data based on the classification result and the first association relationship, the method may further include:

[0016] Obtain the first key performance indicator (KPI) data used to evaluate user service perception;

[0017] Based on the comprehensive assessment results and the target bearer network, determine the abnormal KQI data to be determined in the 5G network, 4G network, and the switching process between the 4G network and the 5G network;

[0018] Inputting the abnormal KQI data to be determined and the first KPI data into a trained relationship calculation model to obtain a first correlation relationship between the abnormal KQI data to be determined and the impact data, wherein the relationship calculation model is a model trained based on historical KPI data and historical KQI data, and the impact data includes at least one first KPI data and a weight corresponding to the first KPI data;

[0019] Determine the KPI data corresponding to the weight greater than or equal to the preset value as the KPI data to be classified;

[0020] Calculate the classification results corresponding to the KPI data to be classified according to the preset binary classification rules;

[0021] According to the classification result and the first association relationship, abnormal KQI data in the abnormal KQI data to be determined is determined.

[0022] In some embodiments, after determining abnormal KQI data in the to-be-determined abnormal KQI data based on the classification result and the first association relationship, the method may further include:

[0023] Determining the abnormal network problem corresponding to the abnormal KQI data based on the correspondence between the KPI data and the network problem and the first association relationship;

[0024] The abnormal network problems corresponding to the abnormal KQI data are sorted into an abnormal network problem list.

[0025] In some embodiments, after organizing the network problems corresponding to the abnormal KQI data into a list of abnormal network problems, the method may further include:

[0026] Obtaining network attribute data associated with the first KPI data;

[0027] A network optimization solution corresponding to at least one abnormal network problem in the abnormal network problem list is determined according to the first KPI data and the network attribute data.

[0028] In some embodiments, after organizing the abnormal network problems corresponding to the abnormal KQI data into an abnormal network problem list, the method may further include:

[0029] Counting the first time series abnormal waveforms corresponding to all abnormal KQI data within a first preset time period;

[0030] Decomposing the first time series waveform according to a preset time series decomposition rule to obtain a second time series abnormal waveform, wherein the second time series abnormal waveform includes at least one peak value and at least one trough value, and the peak value and the trough value correspond to the abnormal KQI data;

[0031] When the peak value and / or the trough value exceeds the first preset range, the time corresponding to the peak value and / or the trough value is determined as the warning time of the abnormal KQI data corresponding to the peak value and / or the trough value;

[0032] According to the warning time and the abnormal KQI data corresponding to the peak value and / or the trough value, warning information within a second preset time period is generated, wherein the first preset time period is earlier than the second preset time period.

[0033] In some embodiments, before inputting the abnormal KQI data to be determined and the first KPI data into the trained relationship calculation model to obtain the first correlation relationship between the abnormal KQI data to be determined and the impact data, the method may further include:

[0034] Obtain historical KPI data and historical KQI data;

[0035] Calculate the loss function corresponding to the to-be-determined weight in the first association relationship based on the historical KPI data and the historical KQI data;

[0036] When the loss function does not meet the training stop condition, the weights to be updated are updated according to the gradient of the loss function and the preset step size until the training stop condition is met, thereby obtaining a trained relationship calculation model.

[0037] In some embodiments, determining the residence factor of the user terminal residing on the 5G network may include:

[0038] For each piece of signaling data, identifying the base station Internet Protocol IP address in the signaling data;

[0039] When the base station IP address is the target base station IP address, calculate the time interval between signaling data corresponding to adjacent target base station IP addresses;

[0040] The time interval duration is determined as the residence factor of the user terminal in the 5G network.

[0041] In some embodiments, when the absolute value of the first correlation indicator is greater than or equal to the first threshold, evaluating the KQI data corresponding to the first correlation indicator according to the 5G network evaluation rule, before obtaining the first evaluation result, the method may further include:

[0042] Obtain the user's historical business perception survey information, business perception dialing information, and historical KQI data distribution information;

[0043] Determine 5G network evaluation rules based on historical users' historical service perception survey information, service perception dialing information, and distribution information of historical KQI data.

[0044] In a second aspect, an embodiment of the present application provides a user service perception evaluation device, the device comprising:

[0045] The first acquisition module is used to acquire signaling data under at least one service for evaluating user service perception.

[0046] The first determination module is used to determine the target bearer network of the KQI data and the residence factor of the user terminal residing in the 5G network based on the signaling data.

[0047] The first calculation module is used to calculate at least one first correlation index for each first KQI data where the target bearer network is a 5G network, based on the first KQI data, the value of the first KQI data under the 4G network, and the Spearman correlation coefficient formula.

[0048] The first evaluation module is used to evaluate the first KQI data according to the 5G network evaluation rule for each first correlation indicator to obtain a first evaluation result when the absolute value of the first correlation indicator is greater than or equal to the first threshold.

[0049] The second evaluation module is configured to evaluate the first KQI data according to the 4G network evaluation rule to obtain a second evaluation result when the absolute value of the first correlation indicator is less than the first threshold.

[0050] The third evaluation module is configured to evaluate the second KQI data for each target bearer network being a 4G network according to a 4G network evaluation rule to obtain a third evaluation result.

[0051] The second calculation module is used to calculate the comprehensive evaluation result of the user's service perception under the service according to the residence factor, the target evaluation result and the third evaluation result.

[0052] In a third aspect, an embodiment of the present application provides a user service perception evaluation device, the device comprising: the device comprising: a processor and a memory storing computer program instructions;

[0053] When the processor executes the computer program instructions, the user service perception evaluation method as described in any embodiment of the present application is implemented.

[0054] In a fourth aspect, an embodiment of the present application provides a computer storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the user service perception evaluation method as described in any embodiment of the present application is implemented.

[0055] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by the processor of the user service perception evaluation device, the user service perception evaluation device performs the user service perception evaluation method as described in any embodiment of the present application.

[0056] A user service perception evaluation method, apparatus, device, medium and program product of an embodiment of the present application obtains signaling data under at least one service for evaluating user service perception, and determines the target bearer network of KQI data and the residence factor of the user terminal residing in the 5G network based on the signaling data. When the bearer network of the KQI data is a 5G network, a first correlation index characterizing the degree of correlation between the KQI data and the 5G network is calculated. Based on the relationship between the absolute value of the first correlation index and the first threshold, a specific network evaluation rule for evaluating the KQI data when the bearer network is a 5G network is determined and the KQI data is evaluated to obtain the first and second evaluation results. The evaluation result of the KQI data when the bearer network is a 4G network and the residence factor are combined with the 4G network evaluation rule to calculate the comprehensive evaluation result of the user service perception. In the embodiment of the present application, based on the first correlation index representing the degree of correlation between KQI data and the 5G network, the degree of dependence between KQI data whose carrier network is a 5G network and the 5G network can be accurately determined, and then whether the 5G network evaluation rules need to be used for evaluation can be accurately determined. Using the 5G network evaluation rules to evaluate KQI data solves the problem in the prior art that it is impossible to evaluate KQI data whose carrier network is a 5G network. In addition, the results of the evaluation of the 4G network and 5G network evaluation rules are combined with the residency factor to comprehensively calculate the user's service perception evaluation results, taking into account the comprehensive impact of the user terminal on service perception when residing in different networks, realizing a comprehensive evaluation of the service perception of 5G users, and accurately reflecting the comprehensive service experience of 5G users in 5G networks and / or 4G networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0058] Figure 1 This is a flow chart of a user service perception evaluation method provided by an embodiment of the present application;

[0059] Figure 2 This is a flow chart of another method for evaluating user service perception provided by an embodiment of the present application;

[0060] Figure 3 This is a flow chart of another method for evaluating user service perception provided by an embodiment of the present application;

[0061] Figure 4 This is a flow chart of another method for evaluating user service perception provided by an embodiment of the present application;

[0062] Figure 5This is a flow chart of another method for evaluating user service perception provided by an embodiment of the present application;

[0063] Figure 6a This is a schematic diagram of an abnormal waveform of a time series in an application scenario provided by an embodiment of the present application;

[0064] Figure 6b This is a schematic diagram of a decomposed time series abnormal waveform in an application scenario provided by an embodiment of the present application;

[0065] Figure 7 This is a schematic diagram of a user service perception evaluation device provided in an embodiment of the present application;

[0066] Figure 8 This is a schematic diagram of a user service perception evaluation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0068] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0069] As described in the background technology, existing service perception methods cannot comprehensively evaluate user service perception in 5G networks.

[0070] The inventors took into consideration that the service perception of 5G users consists of two parts: the service perception of 5G users in the 5G network and the service perception of 5G users when they fall back from the 5G network to the 4G network. The existing service perception evaluation method can evaluate the user service perception of 5G users when they fall back from the 5G network to the 4G network, but it cannot discover and evaluate the user service perception under the 5G network, nor can it evaluate the user's overall service perception when the 5G user switches back and forth between the 4G network and the 5G network. The KQI data used to evaluate user service perception includes two aspects: control plane KQI data and service plane KQI data. The control plane KQI data of the 5G network relies on the 4G network for transmission. Therefore, the existing service perception evaluation method can evaluate the control plane KQI data under the 5G network, but cannot evaluate the service plane KQI data under the 5G network.

[0071] Therefore, the inventors devised a method to "divide" the KQI data for at least one service of a 5G user and evaluate it using existing service perception evaluation methods for both the control and service plane KQI data transmitted over the 4G network. For service plane KQI data on the 5G network, the degree of its dependence on the 5G network is calculated. KQI data with a high degree of dependence is evaluated using the same rules as those used for 5G network evaluation, while KQI data with a low degree of dependence continues to be evaluated using existing service perception evaluation methods. The combined results of these two evaluations yield a comprehensive assessment of 5G users' service perception.

[0072] In order to solve the problems of the prior art, embodiments of the present application provide a user service perception evaluation method, apparatus, device, medium and program product.

[0073] For the convenience of description, the specific process of the user service perception evaluation method is described below with the user service perception evaluation device as the execution subject.

[0074] Figure 1 A schematic diagram of a method for displaying user service perception evaluation provided by an embodiment of the present application is shown, and the method includes:

[0075] S110: Acquire signaling data of at least one service for evaluating user service perception.

[0076] S120: Determine, based on the signaling data, a target bearer network for the KQI data and a residency factor for the user terminal to reside in the 5G network.

[0077] S130, for each first KQI data of the target bearer network being a 5G network, calculate at least one first correlation index according to the first KQI data, the value of the first KQI data in the 4G network, and the Spearman correlation coefficient formula.

[0078] S140, for each first correlation indicator, when the absolute value of the first correlation indicator is greater than or equal to the first threshold, evaluate the first KQI data according to the 5G network evaluation rule to obtain a first evaluation result.

[0079] S150: When the absolute value of the first correlation indicator is less than the first threshold, evaluate the first KQI data according to the 4G network evaluation rule to obtain a second evaluation result.

[0080] S160 : For each second KQI data whose target bearer network is a 4G network, evaluate the second KQI data according to a 4G network evaluation rule to obtain a third evaluation result.

[0081] S170 , calculating a comprehensive evaluation result of the user's service perception under the service according to the residence factor, the target evaluation result, and the third evaluation result.

[0082] In an embodiment of the present application, signaling data under at least one service for evaluating user service perception is obtained, and based on the signaling data, the target bearer network of the KQI data and the residence factor of the user terminal residing in the 5G network are determined. When the bearer network of the KQI data is a 5G network, a first correlation index characterizing the degree of correlation between the KQI data and the 5G network is calculated. Based on the relationship between the absolute value of the first correlation index and the first threshold, a specific network evaluation rule for evaluating the KQI data when the bearer network is a 5G network is determined, and the KQI data is evaluated to obtain the first and second evaluation results. The evaluation result of the KQI data when the bearer network is a 4G network and the residence factor are combined with the 4G network evaluation rule to calculate the comprehensive evaluation result of the user service perception. In the embodiment of the present application, based on the first correlation index representing the degree of correlation between KQI data and the 5G network, the degree of dependence between KQI data whose carrier network is a 5G network and the 5G network can be accurately determined, and then whether the 5G network evaluation rules need to be used for evaluation can be accurately determined. Using the 5G network evaluation rules to evaluate KQI data solves the problem in the prior art that it is impossible to evaluate KQI data whose carrier network is a 5G network. In addition, the results of the evaluation of the 4G network and 5G network evaluation rules are combined with the residency factor to comprehensively calculate the user's service perception evaluation results, taking into account the comprehensive impact of the user terminal on service perception when residing in different networks, realizing a comprehensive evaluation of the service perception of 5G users, and accurately reflecting the comprehensive service experience of 5G users in 5G networks and / or 4G networks.

[0083] In some embodiments, in S110, the signaling data may include at least one of signaling data from a 4G network and signaling data from a 5G network. The KQI data may include control plane KQI data and service plane KQI data. The user service perception evaluation device automatically collects raw signaling data from at least one service in a user dimension, performs data preprocessing on the raw signaling data, and thereby obtains signaling data in a preset data format.

[0084] In some embodiments, the data preprocessing operation may include null value processing, abnormal value processing, and data format conversion. After the user service perception evaluation device obtains the signaling data in a preset data format, it may further include storing the signaling data in a database.

[0085] In one application scenario, the user service perception evaluation device automatically collects signaling data under the 4G network and signaling data under the 5G network under at least one service of the user through the data automatic collection module. Specifically, the data automatic collection module may include a signaling data collection unit, a data preprocessing unit, and a data preprocessing result output unit. The user service perception evaluation device automatically collects signaling data under the service through the signaling data collection unit and stores the collected signaling data in the collection server, and performs data preprocessing operations such as null value processing, abnormal value processing, and data format conversion on the collected signaling data through the data preprocessing unit to obtain signaling data in a preset data format, and then the data preprocessing unit sends the signaling data in the preset data format to the preprocessing result output unit to output the signaling data in the preset data format for subsequent use, and stores the signaling data in the preset data format in the database.

[0086] In some embodiments, the data pre-processing operation may include null value processing, abnormal value processing, and data format conversion. After the acquisition server outputs the signaling data to the user service perception evaluation device, the acquisition server may also store the signaling data in a database.

[0087] The inventors discovered that 5G networks suffer from insufficient coverage in their early stages of construction, severely impacting their ability to access the 5G network and their actual service perception. Therefore, to accurately assess 5G user service perception, it's necessary to accurately assess the 5G residency of 5G user terminals within the 5G network. This invention proposes using signaling data to measure the residency of 5G user terminals within the 5G network.

[0088] In some embodiments, in S120, the target bearer network may include a 4G network and a 5G network, the signaling data includes Internet Protocol Address (IP) address information and bearer network information, and the terminal may include a Mobility Management Entity (MME). The user service perception evaluation device can determine whether the bearer network of the signaling data is a 4G network or a 5G network based on the bearer network information, and further determine whether the bearer network of the KQI data is a 4G network or a 5G network. The IP address information can determine the Internet protocol currently used by the MME and then determine whether the MME uses the 5G network, and then calculate the residency factor of the MME residing in the 5G network.

[0089] In order to more efficiently identify the situation where the user terminal resides on the 5G network, in some embodiments, determining the residency factor of the user terminal residing on the 5G network may include:

[0090] For each piece of signaling data, identify the base station Internet Protocol IP address in the signaling data.

[0091] In some embodiments, when a 5G user conducts services through an MME, the MME exchanges signaling with a Serving GateWay (SGW) and / or a PDN GateWay (PGW). The signaling includes IP addresses of base stations on different networks. The user service perception evaluation device can identify the IP addresses of base stations on different networks through the signaling data.

[0092] When the base station IP address is the target base station IP address, the time interval between signaling data corresponding to adjacent target base station IP addresses is calculated.

[0093] In some embodiments, the target base station IP address may include an IP address for a next-generation NodeB (gNB). When the MME begins utilizing the 5G network, it sends signaling carrying the gNB's IP address to the SGW and / or PGW. When the MME terminates utilizing the 5G network, it also sends signaling carrying the gNB's IP address to the SGW and / or PGW. Furthermore, the user service perception evaluation device can identify at least one set of adjacent signaling data that shares the same gNB IP address and calculate the interval duration between adjacent signaling data that share the same gNB IP address.

[0094] It should be noted that gNB refers to the 5G network bearer base station.

[0095] The time interval duration is determined as the residence factor of the user terminal in the 5G network.

[0096] In some embodiments, the user service perception evaluation device uses the interval duration calculated in the previous step as the residence factor.

[0097] In one application scenario, an MME switches service A from a 4G network to a 5G network and then back to the 4G network. It should be noted that switching from a 4G network cell to a 5G network cell and then back to the 4G network involves three steps: 4G network access, adding a 5G secondary carrier to the 4G network, and releasing the 5G secondary carrier.

[0098] First, the MME performs service A on the 4G network. The Long Term Evolution (LTE) evolved NodeB (eNB) sends signaling to the MME. The MME then sends a "Modify Bearer Request" signaling A1 carrying the eNB's IP address to the SGW and / or PGW, indicating that all future MME signaling will be transmitted based on the eNB pointed to by the IP address. This indicates that the MME has accessed the 4G network.

[0099] The MME then switches from the 4G network to the 5G network to carry out service A until service A is completed. Specifically, the MME switches from the 4G network cell to the 5G network cell. The eNB sends a signaling request to the next-generation NodeB (gNB) to join a 5G secondary carrier. After receiving this signaling, the gNB adds the 5G secondary carrier to the eNB. The MME then sends a "Modify Bearer Request" signaling message carrying the gNB's IP address to the SGW and / or PGW, indicating that all future MME signaling will be transmitted based on the gNB pointed to by this IP address. This indicates that the MME has accessed the 5G network.

[0100] The MME then falls back from the 5G network to the 4G network to carry out service A. Specifically, the MME also sends a "Modify Bearer Request" signaling A2 carrying the gNB's IP address to the SGW and / or PGW. The user service perception evaluation device calculates the interval t1 between signaling A1 and signaling A2 and uses this interval t1 as the MME's residency factor on the 5G network when carrying service A.

[0101] In this embodiment of the present application, the duration between two adjacent signaling data packets whose IP addresses are both the target address is determined as the residency factor of the user terminal on the 5G network. Simply by counting the duration between the occurrences of signaling data packets that meet the conditions, the residency factor reflecting the user terminal's presence on the 5G network can be obtained, without the need for complex calculations or processing, thereby improving the efficiency of identifying the user terminal's presence on the 5G network.

[0102] In some embodiments, in S130, the first correlation index represents the degree of correlation between the first KQI data and the 5G network. The user service perception evaluation device substitutes the KQI data of each 5G network and the value of the KQI data of the 4G network into Formula 1 for each service to calculate the first correlation index.

[0103]

[0104] Among them, ρ represents the first correlation index, x i represents the i-th KQI data of a certain service in the 5G network, n represents the total number of KQI data of a certain service in the 5G network, and y i Indicates the value of the i-th KQI data of a certain service in the 5G network in the 4G network. Indicates the above x i The KQI data represents the average value of the 5G network. Indicates the above x i The KQI data represented by is the average value of the KQI data under the 4G network, where the average value is calculated based on the historical KQI data within the historical time period. The historical time period can be set based on actual experience and is not specifically limited in the embodiments of the present application.

[0105] It can be understood that when the first KQI data does not have a corresponding value under the 4G network, the first association indicator can be set in the range of -1 to 1 according to actual needs. The embodiment of the present application does not limit this. For example, when the first KQI data does not have a corresponding value under the 4G network, the first association indicator can be set to 1.

[0106] As an example, let's take web browsing as the service, and KQI data as page opening delay and first screen delay. When user A browsed the web during a certain period on May xx, xx, the user service perception evaluation device identified that user A's page opening delay under the 5G network was 300ms, and the first screen delay was 500ms; the page opening delay under the 4G network was 400 milliseconds, and the first screen delay was 1000ms. The user service perception evaluation device calculated that the average value of the page opening delay when users browsed the web under the 5G network from January to April in xx was 350ms, and the average value of the first screen delay was 1500ms; the average value of the page opening delay when users browsed the web under the 4G network was 420ms, and the average value of the first screen delay was 2500ms. The user service perception evaluation device substitutes the above data into Formula 1 and calculates the first correlation index ρ to be 0.499. The user service perception evaluation device believes that when user A browses the web during a certain period of time on May xx, xx, the correlation degree between the KQI data page opening delay and the first screen delay and the 5G network is 0.499.

[0107] In one embodiment, in S140, the 5G network evaluation rules may include a 5G perception baseline of the service and a 5G evaluation baseline of abnormal KQI data. The user service perception evaluation device evaluates the first KQI data using the 5G evaluation baseline of the abnormal KQI data for each first associated indicator of the service when the absolute value of the first associated indicator is greater than or equal to a first threshold, to obtain an evaluation result of the first KQI data, and then evaluates the evaluation result of the first KQI data based on the 5G perception baseline of the service to obtain the first evaluation result of the service under the 5G network.

[0108] It should be noted that the first threshold is selected based on experience, and the embodiment of the present application takes 0.2 as an example.

[0109] In order to accurately evaluate the user's perception of the service in the 5G network, in some embodiments, such as Figure 2 The flowchart of another user service perception evaluation method provided by an embodiment of the present application is shown. When the absolute value of the first correlation indicator is greater than or equal to the first threshold, the KQI data corresponding to the first correlation indicator is evaluated according to the 5G network evaluation rule. Before obtaining the first evaluation result, the method may further include S210-S220:

[0110] S210 , respectively obtaining the user's historical service perception survey information, service perception dialing information, and distribution information of historical KQI data.

[0111] In some embodiments, in S210, the service perception dialing test information may include the results of a staff member testing user perception of a specific service using existing dialing test methods. The distribution information of historical KQI data may include the statistical distribution results of historical KQI data in a 5G network divided by user dimensions. The staff member stores the user's historical service perception survey information and service perception dialing test information in a database based on the user dimension. The user service perception evaluation device retrieves the user's historical service perception survey information and service perception dialing test information and historical KQI data from the database based on the user dimension, and performs statistical distribution on the historical KQI data to obtain the distribution information of the historical KQI data.

[0112] S220: Determine 5G network evaluation rules based on historical user service perception survey information, service perception dialing information, and distribution information of historical KQI data.

[0113] In some embodiments, in S220, the user service perception evaluation device integrates the user's historical service perception survey information, service perception dialing information, and distribution information of historical KQI data in the user dimension to determine the 5G evaluation baseline of abnormal KQI data under each service, and integrates the evaluation baseline of each abnormal KQI data to obtain the user's 5G perception baseline for each service, and organizes the 5G perception baseline corresponding to each service into the 5G network evaluation rules.

[0114] As an example, let's take the service as video playback, and the KQI data as video download rate and video freeze frequency. The user's historical video playback perception survey information shows that when the video freeze frequency is 0.5 times / minute or less, the user's service perception is good and the video playback is smooth. When the staff tested the video playback using existing dialing methods, they found that when the video download rate under the 5G network was 1500kbit / s or above, the video played smoothly without obvious freezes. The user service perception evaluation device statistically distributed the captured historical video download rates and video freeze frequencies under the 5G network, and found that 80% of the video download rates were 2000kbit / s or above, and the video freeze frequency was 1 time or less.

[0115] The user service perception evaluation device, after integrating the above information, ultimately determined that the 5G evaluation baseline for measuring video download rates under the 5G network video playback service is greater than or equal to 2000kbit / s, and the evaluation baseline for measuring video freeze frequency is 0.5 times / minute or less. The evaluation baselines for comprehensive video download rate and video freeze frequency determine that the 5G perception baseline for evaluating video playback services is that when the video download rate is greater than or equal to 2000kbit / s and / or the video freeze frequency is less than or equal to 0.5 times / minute, video playback is smooth; when the video download rate is less than 2000kbit / s and / or the video freeze frequency is greater than 0.5 times / minute, video playback is jerky.

[0116] In some embodiments, the user service perception evaluation device can also receive 5G network evaluation rules determined by the staff based on the user's historical service perception survey information, service perception dialing information, and distribution information of historical KQI data combined with actual conditions.

[0117] In the embodiment of the present application, the 5G network evaluation rules are determined by combining a large amount of historical user service perception survey information, service perception dialing test information, and the distribution information of historical KQI data. While considering the user's service perception, the system also assists with the dialing test results of the user's service perception in different situations. The distribution information of the large amount of historical KQI data is combined with the user's service perception results and the dialing test results to achieve a comprehensive and comprehensive baseline for determining the quality of service perception. This in turn provides a 5G network evaluation rule that can accurately evaluate the user's service perception under the 5G network. When the 5G network evaluation rule is used for evaluation, the user's service perception under the 5G network can be accurately evaluated.

[0118] In some embodiments, in S150, the 4G network evaluation rule evaluation may include a service perception evaluation method in the prior art. The user service perception evaluation device evaluates the first KQI data for each first associated indicator of each service using the service perception evaluation method in the prior art when the absolute value of the first associated indicator is greater than or equal to a first threshold, thereby obtaining a user service perception result of the service in the 4G network, i.e., a second evaluation result.

[0119] In some embodiments, in S160, the user service perception evaluation device uses the service perception evaluation method in the prior art to evaluate the second KQI data of the target bearer network being the 4G network, and obtains the user service perception result of a certain service in the 4G network, that is, the third evaluation result.

[0120] In some embodiments, in S170, the target evaluation result is one of the first evaluation result and the second evaluation result. Specifically, for each service, when the service has the first evaluation result, the user service perception evaluation device will calculate the comprehensive evaluation result of the user service perception under the service according to Formula 2. When the service has the second evaluation result, the second evaluation result is merged into the third evaluation result, and the residence factor is set to zero, and the comprehensive evaluation result of the user service perception under the service is still calculated according to Formula 2.

[0121] S=(1-z)*S4+z*S5 Formula 2

[0122] Among them, S represents the comprehensive evaluation result of user service perception, z represents the retention factor, S4 represents the third evaluation result (that is, the user service perception result evaluated using the existing service perception evaluation method), and S4 represents the first evaluation result.

[0123] In some embodiments, the first evaluation result, the second evaluation result, and the third evaluation result may include a service perception score corresponding to the service obtained by the user service perception evaluation device after evaluating the KQI data using corresponding network evaluation rules.

[0124] As an example, take the video playback service, the KQI data is the video download rate and the video freeze frequency. After the user's historical video playback service perception survey information, service perception dialing information and historical KQI data under the 5G network are integrated, the user service perception evaluation device determines that the 5G perception baseline of the video playback service is when the video download rate is greater than or equal to 5000kbit / s and / or the video freeze frequency is less than or equal to 0.1 times / minute, the user service perception score corresponding to the video playback service is 5 points; when the video download rate is less than 5000kbit / s and greater than or equal to 2000kbit / s, and / or the video freeze frequency is greater than When the video download rate is 0.1 times / minute and less than or equal to 0.5 times / minute, the user service perception score corresponding to the video playback service is 2 points; when the video download rate is less than 2000kbit / s and greater than 3500kbit / s, and / or the video freeze frequency is greater than 0.5 times / minute and less than 1 time / minute, the user service perception score corresponding to the video playback service is -2 points; when the video download rate is greater than or equal to 3500kbit / s, and / or the video freeze frequency is greater than or equal to 1 time / minute, the user service perception score corresponding to the video playback service is -5 points.

[0125] In order to improve the efficiency of determining abnormal KQI data, in some embodiments, such as Figure 3 The flowchart of another method for evaluating user service perception provided by an embodiment of the present application is shown. After calculating a comprehensive evaluation result of user service perception under the service based on the residence factor, the target evaluation result, and the third evaluation result, the method may further include S310-S360:

[0126] S310: Acquire first key performance indicator (KPI) data for evaluating user service perception.

[0127] S320: Determine abnormal KQI data to be determined in the 5G network, the 4G network, and the switching process between the 4G network and the 5G network based on the comprehensive evaluation result and the target bearer network.

[0128] S330: Input the abnormal KQI data to be determined and the first KPI data into the trained relationship calculation model to obtain a first correlation relationship between the abnormal KQI data to be determined and the impact data.

[0129] S340: Determine the KPI data corresponding to the weight greater than or equal to the preset value as the KPI data to be classified.

[0130] S350: Calculate the classification result corresponding to the KPI data to be classified according to the preset binary classification rules.

[0131] S360: Determine abnormal KQI data in the abnormal KQI data to be determined based on the classification result and the first association relationship.

[0132] In an embodiment of the present application, the abnormal KQI data to be determined and the first KPI data are input into a trained relationship calculation model to obtain a first correlation relationship between the abnormal KQI data to be determined and the influencing data. The KPI data corresponding to the weight greater than the preset value is determined as the KPI data to be classified, and then the abnormal KQI data is determined based on the classification results corresponding to the KPI data to be classified calculated according to the preset binary classification rules and the first correlation relationship. By automatically calculating the correlation relationship between the KQI data and the KPI data through the model, manual analysis and calculation are avoided, and the calculation efficiency of the correlation relationship between the KQI data and the KPI data is improved. Only the classification results of the KPI data corresponding to the weight greater than the preset value are calculated, which avoids the classification calculation of all KPI data, improves the calculation efficiency of the KPI data, and thus improves the efficiency of judging abnormal KQI data.

[0133] In some embodiments, in S310, the user service perception evaluation device automatically collects base station network management data under at least one service of the user in the user dimension, and performs data preprocessing operations on the base station network management data, thereby obtaining the first KPI data in four preset data formats, including wireless environment, accessibility, retention, and mobility.

[0134] In some embodiments, in S320, the user service perception evaluation device compiles a list of poor-quality users based on the comprehensive evaluation results of user service perception in the above steps, compiles a list of poor-quality services based on the services provided by the poor-quality users, compiles a list of poor-quality cells based on the cells where the poor-quality users reside, and compiles the list of abnormal KQI data to be determined based on the evaluation results of the KQI data in the list of poor-quality services. The abnormal KQI data to be determined is divided into 5G networks, 4G networks, and the abnormal KQI data to be determined during the handover process between 4G networks and 5G networks based on the network carrying the abnormal KQI data to be determined.

[0135] In some embodiments, in S330, the relationship calculation model is a model trained based on historical KPI data and historical KQI data, and the influencing data may include at least one first KPI data and a weight corresponding to the first KPI data. The user service perception evaluation device inputs the quality difference KQI data and the first KPI data under the 5G network in the above step into the trained relationship calculation model, so that the relationship calculation model solves the weight in Formula 3 for each quality difference KQI data according to the gradient descent algorithm, and obtains a first correlation relationship between the quality difference KQI data and the influencing data.

[0136] y=w1x1+w2x2+…+w n x n +b Formula 3

[0137] Among them, y represents KQI data, w1, w 2… w n represents the weight, b represents the bias term, x1, x 2… x n Indicates KPI data associated with KQI data.

[0138] It is understandable that different KQI data may correspond to different deviation terms, and the deviation terms may be pre-set according to actual conditions and solved by the relational calculation model. The specific method for determining KPI data associated with KQI data is prior art and is not specifically limited in the present embodiment.

[0139] It is worth noting that calculating weights (i.e., model parameters) according to the gradient descent algorithm is a prior art and is not specifically limited in the embodiments of the present application. The gradient descent algorithm in the embodiments of the present application includes but is not limited to the batch gradient descent algorithm (Batch Gradient Descent, BGD) or the stochastic gradient descent algorithm (Stochastic Gradient Descent, SGD) or the mini-batch gradient descent algorithm (Mini-batch Gradient Descent, MBGD).

[0140] As an example, take the KQI data of video freeze frequency 0.5 times / minute as an example, and the KPI data related to the video freeze frequency is the channel quality ratio and interference noise. The user service perception evaluation device substitutes the video freeze frequency, channel quality ratio Z and interference noise N into formula 3, and 0.5 times / minute = w 1* Z+w 2* N+b. And according to the gradient descent algorithm, w1 is calculated to be 0.6 and w2 is calculated to be 0.4. Then the first correlation between the KQI data and the KPI data output by the user service perception evaluation device is 0.5 times / minute = 0.6 * Z+0.4 * N+b.

[0141] To improve the efficiency of determining the association relationship between KQI data and KPI data, in some embodiments, before inputting the abnormal KQI data to be determined and the first KPI data into the trained relationship calculation model to obtain the first association relationship between the abnormal KQI data to be determined and the impact data, the method may further include:

[0142] Obtain historical KPI data and historical KQI data.

[0143] In some embodiments, the user service perception evaluation device calls historical KPI data and historical KQI data from a database.

[0144] A loss function corresponding to the to-be-determined weight in the first association relationship is calculated based on the historical KPI data and the historical KQI data.

[0145] In some embodiments, the user service perception evaluation device calculates the loss function corresponding to the weight according to Formula 3 when solving the weight in Formula 3 according to the gradient descent algorithm for each historical KQI data and at least one historical KPI data associated therewith.

[0146] When the loss function does not meet the training stop condition, the weights to be updated are updated according to the gradient of the loss function and the preset step size until the training stop condition is met, thereby obtaining a trained relationship calculation model.

[0147] In some embodiments, the training stop condition may include that the product of the gradient of the current loss function minus the current loss function and the preset step length is less than a second threshold. When the loss function does not meet the training stop condition, the user service perception evaluation device calculates the gradient of the current loss function and the product of the preset step length, and subtracts the gradient of the current loss function and the product of the preset step length from the current weight to be updated to complete the update of the current weight to be updated. When the current loss function meets the training stop condition, the updated weight to be updated is used as the final weight in Formula 3, and Formula 3 is output as the first association relationship.

[0148] In an embodiment of the present application, based on historical KPI data and historical KQI data, a loss function corresponding to the weight to be determined in the first association relationship is calculated. If the loss function does not meet the training stop condition, the gradient and preset step size are used to update the weight to be updated until the training stop condition is met, thereby obtaining a trained relationship calculation model. The condition for stopping model training is that the loss function meets the preset condition, and the iterative update of the weight to be updated is achieved, thereby achieving the update of the first association relationship between the historical KQI data and the historical KPI data, avoiding the multivariate regression calculation of the first association relationship, and improving the efficiency of determining the association relationship between the KQI data and the KPI data.

[0149] In some embodiments, in S340, the user service perception evaluation device determines, for each first association relationship, KPI data corresponding to a weight greater than or equal to a first preset value as KPI data to be classified.

[0150] In some embodiments, the preset value is used to represent the main KPI data that affects the KQI data in the first association relationship. It can be set according to actual conditions. The embodiments of this application do not impose specific restrictions. For example, the preset value can be the value corresponding to the largest weight in the first association relationship.

[0151] In some embodiments, in S350, the preset binary classification rule may include a decision tree algorithm. The user service perception evaluation device performs classification calculation on the KPI data to be classified according to the decision tree algorithm to classify abnormal KPI data and non-abnormal KPI data to obtain a classification result of the KPI data to be classified.

[0152] In some embodiments, the decision tree algorithm may include a binary tree algorithm.

[0153] In some embodiments, in S360, the user service perception evaluation device determines abnormal KPI data based on the classification result of the KPI data to be classified, and determines the KQI data to be determined corresponding to the abnormal KPI data based on the first association relationship and determines the KQI data to be determined as abnormal KQI data.

[0154] In order to identify abnormal network problems in 5G networks, in some embodiments, such as Figure 4 The flowchart of another method for evaluating user service perception provided by an embodiment of the present application is shown. After determining abnormal KQI data in the abnormal KQI data to be determined based on the classification result and the first association relationship, the method may further include S410-S420:

[0155] S410 , determining an abnormal network problem corresponding to the abnormal KQI data according to the corresponding relationship between the KPI data and the network problem, the classification result, and the first association relationship.

[0156] S420: Arrange the abnormal network problems corresponding to the abnormal KQI data into an abnormal network problem list.

[0157] In an embodiment of the present application, after determining the abnormal KQI data, the abnormal KPI data corresponding to the abnormal KQI data can be determined based on the first association relationship and the correspondence between the KPI data network problems, and then the abnormal network problems corresponding to the abnormal KQI data can be determined, and the abnormal network problems can be organized into an abnormal network problem list, thereby realizing the identification of abnormal network problems under the 5G network.

[0158] In some embodiments, in S410, the user service perception evaluation apparatus determines abnormal KPI data based on the classification results of the above steps according to the first association relationship, determines that the abnormal KPI data corresponds to the abnormal KQI data according to the first association relationship, and determines that the abnormal network problem corresponding to the abnormal KPI data is an abnormal network problem corresponding to the abnormal KPI data according to the correspondence between the KPI data and the network problem, and further determines the abnormal network problem as the abnormal network problem corresponding to the abnormal KQI data.

[0159] It should be noted that the specific method for determining the correspondence between KPI data and network problems is an existing technology and is not specifically limited in the embodiments of the present application.

[0160] In some embodiments, in S420, the user service perception evaluation device summarizes abnormal network problems into an abnormal network problem list.

[0161] In one application scenario, the user service perception evaluation device uses a gradient descent algorithm to determine the KPI data that affects user service perception from pre-processed and insightful historical KQI data and historical KPI data, and uses a decision tree algorithm to identify the perception threshold of the KPI data. Based on the model training results of different services, it identifies abnormal network problems under the 5G network and constructs a knowledge base for locating and analyzing abnormal network problems under user browsing, video, instant messaging and other services.

[0162] To improve the optimization efficiency of abnormal network problems, in some embodiments, after organizing the network problems corresponding to the abnormal KQI data into a list of abnormal network problems, the method may further include:

[0163] Obtain network attribute data associated with the first KPI data.

[0164] In some embodiments, the network attribute data may include big data related to network communications that can be associated with KPI data in the prior art. After automatically collecting the first KPI data, the user service perception evaluation device obtains the associated network attribute data.

[0165] A network optimization solution corresponding to at least one abnormal network problem in the abnormal network problem list is determined according to the first KPI data and the network attribute data.

[0166] In some embodiments, the first KPI data includes abnormal KPI data. The user service perception evaluation device determines, for each abnormal network problem, the abnormal KPI data corresponding to the abnormal network problem and its associated network attribute data based on the correspondence between the KPI data and the network problem, and generates a list of at least one optimization solution for optimizing the abnormal network problem.

[0167] As an example, the user service perception evaluation device evaluates the video playback service perception of a group of users in a certain time period and finds that the video playback of this group of users is relatively jerky, and their video playback frequency is relatively high. After the user service perception evaluation device inputs the video playback frequency of the user and the KPI data into the relationship calculation model, the weight of the signal quality difference ratio in the obtained correlation relationship is the largest. Then, the network attribute information associated with the signal quality difference ratio, including base station maintenance information, thermal coverage information, and information on the number of users carried by the base station, is obtained. It is then determined that the reason why the video playback of this group of users is relatively jerky is that the disconnection of the intermediate cell base station has caused the number of users carried by the base station in this cell to exceed the quota. Thus, an optimization plan for the intermediate cell requiring the rectification of the disconnection is generated.

[0168] In this embodiment of the present application, an optimization solution corresponding to an abnormal network problem is determined based on the network attribute data and the second KPI data. The abnormal network problem can then be optimized based on the optimization solution. This automatically generates, builds, maintains, and optimizes the abnormal network problem solution based on the KPI data, avoiding manual analysis and improving the efficiency of optimizing the abnormal network problem.

[0169] In order to achieve early warning of abnormal KQI data, in some embodiments, such as Figure 5 The flowchart of another method for evaluating user service perception provided by an embodiment of the present application is shown. After the abnormal network problems corresponding to the abnormal KQI data are sorted into an abnormal network problem list, the method may further include S510-S540:

[0170] S510 , collecting statistics on first time series abnormal waveforms corresponding to all abnormal KQI data within a first preset time period.

[0171] S520 , decomposing the first time series waveform according to a preset time series decomposition rule to obtain a second time series abnormal waveform.

[0172] S530: When the peak value and / or the trough value exceeds a first preset range, the time corresponding to the peak value and / or the trough value is determined as the warning time of the abnormal KQI data corresponding to the peak value and / or the trough value.

[0173] S540 : Generate warning information within a second preset time period according to the warning time and the abnormal KQI data corresponding to the peak value and / or the trough value.

[0174] In an embodiment of the present application, after the first time series abnormal waveform statistically analyzed within a first preset time period is decomposed according to a preset time series decomposition rule, the warning time of the abnormal KQI data within the first preset time period is determined based on the magnitude relationship between the peak value and / or trough value of the second time series abnormal waveform obtained after decomposition and the first preset range. Warning information for the abnormal KQI data within the second preset time period is then generated based on the warning time and the abnormal KQI data corresponding to the peak value and / or trough value. Based on the time series abnormal waveform, the time when the abnormality may occur within the second preset time period and the KQI data information are predicted, thereby achieving early warning of abnormal KQI data.

[0175] In some embodiments, in S510, the user service perception evaluation device performs statistics on abnormal KQI data according to a time series algorithm within a first preset time period to obtain a first time series abnormal waveform.

[0176] In some embodiments, in S520, the second time series abnormal waveform may include at least one peak value and at least one trough value, and the peak value and the trough value correspond to the abnormal KQI data. The second time series abnormal waveform may include at least one of a long-term trend waveform, a seasonal fluctuation waveform, and a random fluctuation waveform. The preset time series decomposition rule may include an additive model (Formula 4) or a multiplicative model (Formula 5) of the time series. The user service perception evaluation device decomposes the first time series abnormal waveform according to the additive model or the multiplicative model to obtain a decomposed second time series abnormal waveform.

[0177] Y(t)=T(t)+S(t)+R(t) Formula 4,

[0178] Y(t)=T(t)*S(t)*R(t) Formula 5,

[0179] Among them, Y(t) represents the abnormal waveform of the first time series, T(t) represents the long-term trend waveform, S(t) represents the seasonal fluctuation waveform, R(t) represents the random fluctuation waveform, and t represents time.

[0180] It should be noted that the first preset time period can be set according to actual needs and is not specifically limited in the embodiments of the present application. The second preset time period corresponds to the first preset time period and is later than the first preset time period. For example, the first preset time period is January to March 2020, and the second preset time period is January to March 2021.

[0181] As an example, the user service perception evaluation device counts the video freeze frequency of user C in the first preset time period from 0:00 on September 14, 2020 to 14:00 on September 27, 2020 according to the time series algorithm, and obtains the following: Figure 6a The first time series abnormal waveform is decomposed to obtain the following Figure 6b The random fluctuation waveform is shown.

[0182] In some embodiments, in S530, the first preset range is -4σ to +4σ or -3σ to +3σ determined according to the normal distribution graph. When the peak value and / or trough value of the abnormal waveform of the second time series exceeds the first preset range, the user service perception evaluation device determines the time corresponding to the peak value and / or trough value as the warning time of the abnormal KQI data corresponding to the peak value and / or trough value.

[0183] As an example, see Figure 6b , if the value of peak 601 exceeds +3σ, the time corresponding to peak 601, 21:00 on September 23, 2020, is determined as the warning time of the video freeze frequency corresponding to the peak value and / or trough value.

[0184] In some embodiments, in S540, the first preset time period is earlier than the second preset time period. The user service perception evaluation device uses the warning time within the first preset time period as the warning time within the second preset time period, and uses the abnormal KQI data corresponding to the peak value and / or trough value of the first preset time period as the abnormal KQI data within the second preset time period, and generates warning information indicating that abnormal KQI data will occur within the second preset time period at the warning time.

[0185] As an example, continuing to refer to the above example, the video freeze frequency 601 of user C deteriorated to 0.23, exceeding +3σ at 21:00 on September 23, 2020, and the user service perception evaluation device generated an early warning information that the video freeze frequency of user C deteriorated at 21:00 on September 23, 2021.

[0186] In some embodiments, the user service perception evaluation device can also count the second time series abnormal waveform corresponding to the poor quality service list, poor quality cell list, poor quality user list or abnormal KPI data within the first preset time period. According to the preset time series decomposition rule, the second time series abnormal waveform is decomposed to obtain a third time series abnormal waveform, wherein the third time series abnormal waveform includes at least one peak value and at least one trough value, and the peak value and the trough value correspond to the poor quality service, poor quality cell, poor quality user or abnormal KPI data. When the peak value and / or the trough value exceed the first preset range, the time corresponding to the peak value and / or the trough value is determined as the warning time of the poor quality service, poor quality cell, poor quality user or abnormal KPI data. According to the warning time and the poor quality service, poor quality cell, poor quality user or abnormal KPI data corresponding to the peak value and / or the trough value, the warning information within the second preset time period is generated.

[0187] Based on the user service perception evaluation method provided in any of the above embodiments, this application also provides an embodiment of a user service perception evaluation device, see Figure 7 .

[0188] Figure 7 FIG. 1 shows a schematic diagram of a user service perception evaluation device provided by an embodiment of the present application. Figure 7 As shown, the user service perception evaluation device 700 may include:

[0189] The first acquisition module 710 is configured to acquire signaling data of at least one service for evaluating user service perception.

[0190] The first determination module 720 is used to determine the target bearer network of the KQI data and the residence factor of the user terminal residing in the 5G network based on the signaling data.

[0191] The first calculation module 730 is used to calculate at least one first correlation index for each first KQI data where the target bearer network is a 5G network, based on the first KQI data, the value of the first KQI data under the 4G network, and the Spearman correlation coefficient formula.

[0192] The first evaluation module 740 is used to evaluate the first KQI data according to the 5G network evaluation rule for each first correlation indicator to obtain a first evaluation result when the absolute value of the first correlation indicator is greater than or equal to the first threshold.

[0193] The second evaluation module 750 is configured to evaluate the first KQI data according to the 4G network evaluation rule to obtain a second evaluation result when the absolute value of the first correlation indicator is less than the first threshold.

[0194] The third evaluation module 760 is configured to evaluate the second KQI data for each target bearer network being a 4G network according to the 4G network evaluation rule to obtain a third evaluation result.

[0195] The second calculation module 770 is used to calculate a comprehensive evaluation result of the user's service perception under the service according to the residence factor, the target evaluation result and the third evaluation result.

[0196] The device in the embodiment of the present application obtains signaling data under at least one service for evaluating user service perception, and determines the target bearer network of the KQI data and the residence factor of the user terminal residing in the 5G network based on the signaling data. When the bearer network of the KQI data is a 5G network, a first correlation index characterizing the degree of correlation between the KQI data and the 5G network is calculated. Based on the relationship between the absolute value of the first correlation index and the first threshold, a specific network evaluation rule for evaluating the KQI data when the bearer network is a 5G network is determined and the KQI data is evaluated to obtain the first and second evaluation results. The evaluation result of the KQI data when the bearer network is a 4G network and the residence factor are combined with the 4G network evaluation rule to calculate the comprehensive evaluation result of the user service perception. In the embodiment of the present application, based on the first correlation index representing the degree of correlation between KQI data and the 5G network, the degree of dependence between KQI data whose carrier network is a 5G network and the 5G network can be accurately determined, and then whether the 5G network evaluation rules need to be used for evaluation can be accurately determined. Using the 5G network evaluation rules to evaluate KQI data solves the problem in the prior art that it is impossible to evaluate KQI data whose carrier network is a 5G network. In addition, the results of the evaluation of the 4G network and 5G network evaluation rules are combined with the residency factor to comprehensively calculate the user's service perception evaluation results, taking into account the comprehensive impact of the user terminal on service perception when residing in different networks, realizing a comprehensive evaluation of the service perception of 5G users, and accurately reflecting the comprehensive service experience of 5G users in 5G networks and / or 4G networks.

[0197] In some embodiments, in order to improve the efficiency of determining abnormal KQI data, the user service perception evaluation device 700 may further include:

[0198] The second acquisition module is used to acquire first key performance indicator KPI data for evaluating user service perception.

[0199] The second determination module is used to determine the abnormal KQI data to be determined in the 5G network, the 4G network, and the switching process between the 4G network and the 5G network based on the comprehensive evaluation results and the target bearer network.

[0200] The input module is used to input the abnormal KQI data to be determined and the first KPI data into the trained relationship calculation model to obtain a first correlation relationship between the abnormal KQI data to be determined and the impact data.

[0201] The third determining module is configured to determine KPI data corresponding to weights greater than or equal to a preset value as KPI data to be classified.

[0202] The third calculation module is used to calculate the classification result corresponding to the KPI data to be classified according to the preset binary classification rules.

[0203] The fourth determining module is configured to determine abnormal KQI data in the abnormal KQI data to be determined based on the classification result and the first association relationship.

[0204] The device in the embodiment of the present application inputs the abnormal KQI data to be determined and the first KPI data into the trained relationship calculation model to obtain the first correlation relationship between the abnormal KQI data to be determined and the influencing data. The KPI data corresponding to the weight greater than the preset value is determined as the KPI data to be classified, and then the abnormal KQI data is determined according to the classification result corresponding to the KPI data to be classified calculated according to the preset binary classification rule and the first correlation relationship. By automatically calculating the correlation relationship between KQI data and KPI data through the model, manual analysis and calculation are avoided, and the calculation efficiency of the correlation relationship between KQI data and KPI data is improved. Only the classification results of the KPI data corresponding to the weight greater than the preset value are calculated, which avoids the classification calculation of all KPI data, improves the calculation efficiency of KPI data, and thus improves the efficiency of judging abnormal KQI data.

[0205] In some embodiments, in order to identify abnormal network problems in a 5G network, the user service perception evaluation device 700 may further include:

[0206] The fifth determining module is configured to determine the abnormal network problem corresponding to the abnormal KQI data according to the corresponding relationship between the KPI data and the network problem, the classification result, and the first association relationship.

[0207] The sorting module is used to sort the abnormal network problems corresponding to the abnormal KQI data into an abnormal network problem list.

[0208] The device in the embodiment of the present application, after determining the abnormal KQI data, can determine the abnormal KPI data corresponding to the abnormal KQI data based on the first association relationship and the correspondence between the KPI data network problems, and then can determine the abnormal network problems corresponding to the abnormal KQI data, and organize the abnormal network problems into an abnormal network problem list, thereby realizing the identification of abnormal network problems under the 5G network.

[0209] In some embodiments, in order to improve the optimization efficiency of abnormal network problems, the user service perception evaluation device 700 may further include:

[0210] The third acquisition module is used to acquire network attribute data associated with the first KPI data.

[0211] The sixth determining module is configured to determine a network optimization solution corresponding to at least one abnormal network problem in the abnormal network problem list according to the first KPI data and the network attribute data.

[0212] The device in the embodiment of the present application determines an optimization solution corresponding to an abnormal network problem based on network attribute data and second KPI data. The abnormal network problem can then be optimized based on the optimization solution. This automatically generates, builds, maintains, and optimizes the abnormal network problem using KPI data, avoiding manual analysis and improving the efficiency of optimizing the abnormal network problem.

[0213] In some embodiments, in order to achieve early warning of abnormal KQI data, the user service perception evaluation device 700 may further include:

[0214] The statistics module is used to count the first time series abnormal waveforms corresponding to all abnormal KQI data within a first preset time period.

[0215] The decomposition module is used to decompose the first time series waveform according to a preset time series decomposition rule to obtain a second time series abnormal waveform.

[0216] The sixth determining module is configured to determine the time corresponding to the peak value and / or trough value as the warning time of the abnormal KQI data corresponding to the peak value and / or trough value when the peak value and / or trough value exceeds the first preset range.

[0217] The generating module is configured to generate warning information within a second preset time period according to the warning time and the abnormal KQI data corresponding to the peak value and / or the trough value.

[0218] The device in the embodiment of the present application decomposes the first time series abnormal waveform statistically collected within the first preset time period according to a preset time series decomposition rule, and then determines the warning time of abnormal KQI data within the first preset time period based on the relationship between the peak value and / or trough value of the second time series abnormal waveform obtained after decomposition and the first preset range. Then, based on the warning time and the abnormal KQI data corresponding to the peak value and / or trough value, warning information of the abnormal KQI data within the second preset time period is generated. Based on the abnormal time series waveform, the time when the abnormality may occur and the KQI data information within the second preset time period are predicted, thereby achieving early warning of abnormal KQI data.

[0219] In some embodiments, in order to improve the efficiency of determining the association relationship between KQI data and KPI data, the user service perception evaluation device 700 may further include:

[0220] The fourth acquisition module is used to acquire historical KPI data and historical KQI data.

[0221] The fourth calculation module is used to calculate the loss function corresponding to the to-be-determined weight in the first association relationship according to the historical KPI data and the historical KQI data.

[0222] The update module is used to update the weights to be updated according to the gradient of the loss function and the preset step size when the loss function does not meet the training stop condition, so as to obtain the trained relationship calculation model.

[0223] The device in the embodiment of the present application calculates the loss function corresponding to the weight to be determined in the first association relationship based on historical KPI data and historical KQI data. When the loss function does not meet the training stop condition, the gradient and preset step size are used to update the weight to be updated until the training stop condition is met, thereby obtaining a trained relationship calculation model. The condition for stopping model training is that the loss function meets the preset condition, and the iterative update of the weight to be updated is achieved, thereby achieving the update of the first association relationship between the historical KQI data and the historical KPI data, avoiding the multivariate regression calculation of the first association relationship, and improving the efficiency of determining the association relationship between the KQI data and the KPI data.

[0224] In some embodiments, in order to more efficiently identify the situation where the user terminal resides on the 5G network, the determination module 720 may include:

[0225] The identification unit is used to identify the base station Internet Protocol IP address in the signaling data for each piece of signaling data.

[0226] The calculation unit is used to calculate the time interval between signaling data corresponding to adjacent target base station IP addresses when the base station IP address is the target base station IP address.

[0227] A determination unit is used to determine the time interval duration as a residence factor for the user terminal to reside in the 5G network.

[0228] The device in the embodiment of the present application determines the duration between two adjacent signaling data with the same IP address as the target address as the resident factor of the user terminal on the 5G network. Simply by counting the duration between signaling data that meet the conditions, the resident factor reflecting the user terminal's 5G network presence can be obtained, eliminating the need for complex calculations or processing, thereby improving the efficiency of identifying the user terminal's 5G network presence.

[0229] In some embodiments, in order to accurately evaluate the user's perception of the service in the 5G network, the first evaluation module 740 may further include:

[0230] The acquisition submodule is used to obtain the user's historical business perception survey information, business perception dialing information and historical KQI data distribution information.

[0231] The determination submodule is used to determine the 5G network evaluation rules based on the historical service perception survey information of historical users, service perception dialing information and the distribution information of historical KQI data.

[0232] In the device of the embodiment of the present application, the 5G network evaluation rules are comprehensively determined by a large amount of historical user service perception survey information, service perception dialing information, and the distribution information of historical KQI data. While considering the user's service perception, the system also assists with the dialing test results of the user's service perception in different situations, and integrates the distribution information of a large amount of historical KQI data with the user's service perception results and dialing test results to achieve a comprehensive and comprehensive baseline for determining the quality of service perception. This in turn obtains a 5G network evaluation rule that can accurately evaluate the user's service perception under the 5G network. When the 5G network evaluation rule is used for evaluation, the user's service perception under the 5G network can be accurately evaluated.

[0233] In addition, in combination with the data storage and data display methods of the above embodiments, Figure 8 As shown, an embodiment of the present application may provide a user service perception evaluation device, which may include a processor 810 and a memory 820 storing computer program instructions.

[0234] Specifically, the processor 810 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0235] The memory 820 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 820 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 820 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 820 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 820 is a non-volatile solid-state memory. In a specific embodiment, the memory 820 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0236] The processor 810 implements any one of the user service perception evaluation methods in the above embodiments by reading and executing computer program instructions stored in the memory 820 .

[0237] In one example, the user service perception evaluation device may further include a communication interface 830 and a bus 840. As shown in FIG6, the processor 810, the memory 820, and the communication interface 830 are connected via the bus 840 and communicate with each other.

[0238] The communication interface 830 is mainly used to implement communication between various modules, devices, units and / or devices in the embodiments of the present application.

[0239] Bus 840 comprises hardware, software or both, and the parts of user business perception assessment equipment are coupled to each other.For example, and not limitation, bus can comprise accelerated graphics port (AGP) or other graphics bus, enhanced industry standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industry standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations.In suitable cases, bus 840 can comprise one or more buses.Although the present application embodiment describes and shows specific bus, the application considers any suitable bus or interconnection.

[0240] When the user service perception evaluation device executes the computer program instructions, the user service perception evaluation method described in any one of the above embodiments is implemented.

[0241] In addition, in combination with the above-mentioned user service perception evaluation method, an embodiment of the present application may provide a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the user service perception evaluation method described in any of the above-mentioned embodiments is implemented.

[0242] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0243] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0244] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0245] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0246] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A user service perception evaluation method, characterized in that: The method comprises: Acquire signaling data under at least one service for evaluating user service perception, wherein the signaling data includes at least one key quality indicator (KQI) data; Determining, according to the signaling data, a target bearer network for the KQI data and a residency factor for the user terminal to reside in a 5G network, wherein the target bearer network includes a 4G network and a 5G network; For each first KQI data whose target bearer network is a 5G network, calculating at least one first correlation index based on the first KQI data, a value of the first KQI data under the 4G network, and a Spearman correlation coefficient formula, wherein the first correlation index represents a degree of correlation between the first KQI data and the 5G network; For each first correlation indicator, when the absolute value of the first correlation indicator is greater than or equal to a first threshold, evaluating the first KQI data according to a 5G network evaluation rule to obtain a first evaluation result; When the absolute value of the first correlation indicator is less than the first threshold, evaluating the first KQI data according to a 4G network evaluation rule to obtain a second evaluation result; For each second KQI data where the target bearer network is a 4G network, evaluating the second KQI data according to the 4G network evaluation rule to obtain a third evaluation result; A comprehensive evaluation result of user service perception under the service is calculated based on the retention factor, the target evaluation result and the third evaluation result, wherein the target evaluation result is one of the first evaluation result and the second evaluation result.

2. The method according to claim 1, characterized in that After calculating a comprehensive evaluation result of user service perception under the service based on the residence factor, the target evaluation result, and the third evaluation result, the method further includes: Obtain the first key performance indicator (KPI) data used to evaluate user service perception; Determining, based on the comprehensive evaluation result and the target bearer network, abnormal KQI data to be determined in the 5G network, the 4G network, and the switching process between the 4G network and the 5G network; Inputting the abnormal KQI data to be determined and the first KPI data into a trained relationship calculation model to obtain a first correlation relationship between the abnormal KQI data to be determined and the impact data, wherein the relationship calculation model is a model trained based on historical KPI data and historical KQI data, and the impact data includes at least one first KPI data and a weight corresponding to the first KPI data; Determine the KPI data corresponding to the weight greater than or equal to the preset value as the KPI data to be classified; Calculate the classification result corresponding to the KPI data to be classified according to the preset binary classification rules; Abnormal KQI data in the abnormal KQI data to be determined is determined according to the classification result and the first association relationship.

3. The method according to claim 2, characterized in that After determining abnormal KQI data in the abnormal KQI data to be determined based on the classification result and the first association relationship, the method further includes: Determining the abnormal network problem corresponding to the abnormal KQI data based on the correspondence between the KPI data and the network problem, the classification result, and the first association relationship; The abnormal network problems corresponding to the abnormal KQI data are sorted into an abnormal network problem list.

4. The method according to claim 2, characterized in that After organizing the network problems corresponding to the abnormal KQI data into a list of abnormal network problems, the method further includes: Obtaining network attribute data associated with the first KPI data; A network optimization solution corresponding to at least one abnormal network problem in the abnormal network problem list is determined based on the first KPI data and the network attribute data.

5. The method according to claim 4, characterized in that After arranging the abnormal network problems corresponding to the abnormal KQI data into an abnormal network problem list, the method further includes: Counting the first time series abnormal waveforms corresponding to all the abnormal KQI data within a first preset time period; Decomposing the first time series waveform according to a preset time series decomposition rule to obtain a second time series abnormal waveform, wherein the second time series abnormal waveform includes at least one peak value and at least one trough value, and the peak value and the trough value correspond to abnormal KQI data; When the peak value and / or the trough value exceeds a first preset range, determining the time corresponding to the peak value and / or the trough value as the warning time of the abnormal KQI data corresponding to the peak value and / or the trough value; According to the warning time and the abnormal KQI data corresponding to the peak value and / or the trough value, warning information within a second preset time period is generated, wherein the first preset time period is earlier than the second preset time period.

6. The method according to claim 2, characterized in that Before inputting the abnormal KQI data to be determined and the first KPI data into the trained relationship calculation model to obtain the first correlation relationship between the abnormal KQI data to be determined and the impact data, the method further includes: Acquire the historical KPI data and the historical KQI data; Calculating a loss function corresponding to the weight to be updated in the first association relationship based on the historical KPI data and the historical KQI data; When the loss function does not meet the training stop condition, the weights to be updated are updated according to the gradient of the loss function and a preset step size until the training stop condition is met, thereby obtaining a trained relationship calculation model.

7. The method according to claim 1, characterized in that Determining the residency factor of the user terminal residing on the 5G network specifically includes: For each piece of signaling data, identifying a base station Internet Protocol (IP) address in the signaling data; When the base station IP address is the target base station IP address, calculating the time interval between signaling data corresponding to adjacent target base station IP addresses; The duration of the time interval is determined as the residence factor of the user terminal in the 5G network.

8. The method according to claim 1, characterized in that When the absolute value of the first correlation indicator is greater than or equal to a first threshold, evaluating the KQI data corresponding to the first correlation indicator according to the 5G network evaluation rule, before obtaining a first evaluation result, the method further includes: Obtain the user's historical business perception survey information, business perception dialing information, and historical KQI data distribution information; The 5G network evaluation rules are determined based on the user's historical business perception survey information, business perception dialing information, and distribution information of historical KQI data.

9. A user service perception evaluation device, characterized in that: The device comprises: A first acquisition module is configured to acquire signaling data under at least one service for evaluating user service perception, wherein the signaling data includes at least one key quality indicator (KQI) data; A first determining module is configured to determine, based on the signaling data, a target bearer network for the KQI data and a residency factor for the user terminal to reside in a 5G network, wherein the target bearer network includes a 4G network and a 5G network; a first calculation module, configured to calculate, for each first KQI data where the target bearer network is a 5G network, at least one first correlation index based on the first KQI data, a value of the first KQI data under a 4G network, and a Spearman correlation coefficient formula, wherein the first correlation index represents a degree of correlation between the first KQI data and the 5G network; a first evaluation module, configured to, for each first correlation indicator, evaluate the first KQI data according to a 5G network evaluation rule to obtain a first evaluation result when an absolute value of the first correlation indicator is greater than or equal to a first threshold; a second evaluation module, configured to evaluate the first KQI data according to a 4G network evaluation rule to obtain a second evaluation result when the absolute value of the first correlation indicator is less than the first threshold; A third evaluation module is configured to evaluate, for each second KQI data whose target bearer network is a 4G network, the second KQI data according to the 4G network evaluation rule to obtain a third evaluation result; The second calculation module is used to calculate the comprehensive evaluation result of user service perception under the service based on the retention factor, the target evaluation result and the third evaluation result, wherein the target evaluation result is one of the first evaluation result and the second evaluation result.

10. A user service perception evaluation device, characterized in that: The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method according to any one of claims 1 to 8 is implemented.

11. A computer storage medium, characterized in that The computer storage medium stores computer program instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of a user service awareness evaluation device, the user service awareness evaluation device is caused to perform the method according to any one of claims 1 to 8.

Images