5g private network sensing evaluation method and device, electronic equipment and readable storage medium

Abstract

The application provides a 5G private network perception evaluation method and device, electronic equipment and readable storage medium. The method comprises the following steps: collecting real-time signaling plane data and real-time user plane data of a to-be-evaluated service in a preset time period; obtaining a first key indicator set corresponding to a network support layer and a second key indicator set corresponding to a general performance layer according to the real-time signaling plane data; determining a third key indicator set corresponding to a service scenario layer according to the real-time user plane data and a preset application scenario feature library; constructing a three-layer perception evaluation model according to the first key indicator set, the second key indicator set and the third key indicator set; and calculating a comprehensive perception evaluation score according to the perception evaluation model, a preset threshold score and a weight parameter. The method improves the efficiency and quality of 5G private network perception evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of 5G, in particular to a 5G private network perception evaluation method and device, electronic equipment and readable storage medium. BACKGROUND

[0002] With the help of intelligent and digital transformation and upgrading of production methods, the construction of 5G private networks of enterprises is also accelerating. 5G private networks carry various application scenarios with technical characteristics such as large bandwidth, low latency, high reliability, and massive connections. Compared with traditional private networks, the network quality requirements are more stringent. Due to the distinct characteristics of diversified business scenarios and differentiated business needs of 5G private networks, how to accurately evaluate the quality of 5G private networks and user perception has become a major challenge.

[0003] At present, it is common to analyze the network quality of 5G public user general multimedia services such as high-definition video, games, web access, rely on manual mode combined with business characteristics for analysis and deduction, and then calculate the perception score according to the manually set index weight. This method needs to manually identify general application scenarios, which is slow, inefficient, and highly dependent on personnel. However, since general business is relatively mature, it can still have good perception evaluation effect.

[0004] However, the application scenarios of vertical industries are diverse, and new applications and new scenarios are emerging. It is extremely difficult to analyze the characteristics of each type of application scenario one by one through manual analysis. Not only does the accuracy depend on the expertise of experts, but it is also difficult to closely match the real use of users. Therefore, it is necessary to automatically and accurately identify 5G private network application scenarios to improve the efficiency and quality of 5G private network perception evaluation. SUMMARY

[0005] The present application provides a 5G private network perception evaluation method, device, electronic equipment and readable storage medium to solve the problem of how to improve the efficiency and quality of 5G private network perception evaluation.

[0006] In a first aspect, the present application provides a 5G private network perception evaluation method, comprising:

[0007] Collecting real-time signaling plane data and real-time user plane data of the business to be evaluated within a preset time period;

[0008] According to the real-time signaling plane data, obtaining a first set of key indicators corresponding to the network support layer and a second set of key indicators corresponding to the general performance layer;

[0009] According to the real-time user plane data and the preset application scenario feature library, determining a third set of key indicators corresponding to the business scenario layer;

[0010] construct a three-layer perception evaluation model according to the first key indicator set, the second key indicator set, and the third key indicator set;

[0011] According to the perception evaluation model and the preset threshold score, a perception evaluation comprehensive score is calculated in combination with a weight parameter.

[0012] Optionally, before the determining the third key indicator set corresponding to the service scenario layer according to the real-time user plane data and the preset application scenario feature library, the method further includes:

[0013] Full-volume collection of signaling offline data and user plane offline data as sample data;

[0014] Performing clustering operation on the sample data through unsupervised learning to output a feature vector;

[0015] Constructing a mapping relationship among an application scenario, the feature vector, and the third key indicator set, and outputting the preset application scenario feature library.

[0016] Optionally, the constructing the mapping relationship among the application scenario, the feature vector, and the third key indicator set includes:

[0017] Labeling the feature vector and dividing it into a corresponding application scenario;

[0018] Establishing the third key indicator set for each application scenario to construct the mapping relationship among the application scenario, the feature vector, and the third key indicator set.

[0019] Optionally, the determining the third key indicator set corresponding to the service scenario layer according to the real-time user plane data and the preset application scenario feature library includes:

[0020] Performing format processing on the real-time user plane data to form a real-time feature vector;

[0021] Performing similarity calculation on the real-time feature vector and a feature vector in the preset application scenario feature library;

[0022] Determining an application scenario corresponding to the real-time user plane data to output the third key indicator set corresponding to the service scenario layer.

[0023] Optionally, the determining the application scenario corresponding to the real-time user plane data includes:

[0024] If the similarity is greater than or equal to a preset threshold, determining the application scenario corresponding to the real-time user plane data according to the preset application scenario feature library;

[0025] If the similarity is less than a preset threshold, an application scenario is recommended according to a preset algorithm, and the real-time feature vector is labeled to be supplemented into the preset application scenario feature library.

[0026] Optionally, the calculating the comprehensive score of the perception evaluation in combination with the weight parameter according to the perception evaluation model and the preset threshold score includes:

[0027] The network support layer, the general performance layer, the service scenario layer, and the first key indicator set, the second key indicator set, and the third key indicator set are configured with different weight parameters and preset thresholds.

[0028] The calculating the comprehensive score of the perception evaluation in combination with the weight parameter according to the perception evaluation model and the preset threshold score.

[0029] Optionally, the 5G private network perception evaluation method further includes:

[0030] The perception evaluation result is presented, and a quality difference index is output, the perception evaluation result being determined according to the comprehensive score of the perception evaluation, and the quality difference index being an index that needs to be optimized.

[0031] In a second aspect, the application provides a 5G private network perception evaluation device, including:

[0032] The acquisition module is configured to collect real-time signaling plane data and real-time user plane data of a service to be evaluated in a preset time period.

[0033] The processing module is configured to acquire a first key indicator set corresponding to a network support layer and a second key indicator set corresponding to a general performance layer according to the real-time signaling plane data, and determine a third key indicator set corresponding to a service scenario layer according to the real-time user plane data and a preset application scenario feature library.

[0034] The model establishing module is configured to construct a three-layer perception evaluation model according to the first key indicator set, the second key indicator set, and the third key indicator set.

[0035] The evaluation module is configured to calculate a comprehensive score of the perception evaluation in combination with a weight parameter according to a perception evaluation model and a preset threshold score.

[0036] Optionally, the 5G private network perception evaluation device further includes a training module.

[0037] The training module is configured to collect full-volume signaling offline data and user plane offline data as sample data before determining the third key indicator set corresponding to the service scenario layer according to the real-time user plane data and a preset application scenario feature library; perform clustering operation on the sample data through unsupervised learning to output a feature vector; and construct a mapping relationship among the application scenario, the feature vector, and the third key indicator set, and output the preset application scenario feature library.

[0038] Optionally, the training module is further configured to label the feature vector and divide it into a corresponding application scenario; and establish the third key indicator set for each application scenario to construct the mapping relationship among the application scenario, the feature vector, and the third key indicator set.

[0039] Optionally, the processing module is further configured to perform format processing on the real-time user plane data to form a real-time feature vector; perform similarity calculation on the real-time feature vector and a feature vector in the preset application scenario feature library; determine the application scenario corresponding to the real-time user plane data to output the third key indicator set corresponding to the service scenario layer.

[0040] Optionally, the processing module is further configured to, if the similarity is greater than or equal to a preset threshold, determine the application scenario corresponding to the real-time user plane data according to the preset application scenario feature library.

[0041] If the similarity is less than the preset threshold, recommend an application scenario according to a preset algorithm, and label the real-time feature vector to supplement into the preset application scenario feature library.

[0042] Optionally, the evaluation module is further configured to configure different weight parameters and preset thresholds for the network support layer, the general performance layer, the service scenario layer, and the first key indicator set, the second key indicator set, and the third key indicator set corresponding thereto; score according to the perception evaluation model and the preset threshold, and calculate the perception evaluation comprehensive score in combination with the weight parameters.

[0043] Optionally, the 5G private network perception evaluation device further comprises a presentation module.

[0044] The presentation module is configured to present a perception evaluation result and output a quality difference indicator, wherein the perception evaluation result is determined according to the perception evaluation comprehensive score, and the quality difference indicator is an indicator that needs to be optimized.

[0045] In a third aspect, the present application provides an electronic device, comprising a processor and a memory connected with the processor in communication;

[0046] The memory stores computer execution instructions.

[0047] The processor executes computer-executed instructions stored in the memory to implement the 5G private network perception evaluation method as described above.

[0048] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer-executed instructions, and the computer-executed instructions are executed by a processor to implement the 5G private network perception evaluation method as described above.

[0049] The 5G private network perception evaluation method provided by the present application collects real-time signaling plane data and real-time user plane data of a business to be evaluated within a preset time period; obtains a first set of key indicators corresponding to a network support layer and a second set of key indicators corresponding to a general performance layer according to the real-time signaling plane data; and determines a third set of key indicators corresponding to a business scenario layer according to the real-time user plane data and a preset application scenario feature library; then constructs a three-layer perception evaluation model according to the first set of key indicators, the second set of key indicators, and the third set of key indicators; and calculates a comprehensive perception evaluation score according to the perception evaluation model, a preset threshold, and a weight parameter. On the one hand, the third set of key indicators corresponding to the business scenario layer is determined by comparing the real-time user plane data and the preset application scenario feature library, without the need for manual identification of the application scenario, so that the identification speed is fast and the efficiency is high, and the method no longer depends on the technology and experience of experts. On the other hand, by obtaining the signaling plane data and the user plane data of the business to be evaluated, the sets of key indicators corresponding to the network support layer, the general performance layer, and the business scenario layer are determined, and there is a corresponding three-layer perception evaluation model for each business to be evaluated, which includes quality perception analysis of the upper business establishment and interaction scenario, so that the 5G private network perception evaluation obtained in this way is more accurate, fully considers different needs in different business scenarios, and can provide better, more optimal, and more accurate services for 5G private network customers, thereby improving the efficiency and quality of 5G private network perception evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0051] Figure 1 A flowchart of a 5G private network perception evaluation method provided by an embodiment of the present application is shown in the figure.

[0052] Figure 2 A structural diagram of a perception evaluation model provided by an embodiment of the present application is shown in the figure.

[0053] Figure 3 A flowchart of a 5G private network perception evaluation method provided by another embodiment of the present application is shown in the figure.

[0054] Figure 4A schematic diagram of an application scenario feature recognition provided for the present application;

[0055] Figure 5 A schematic diagram of a preset application scenario feature library provided for the present application;

[0056] Figure 6 A flowchart of a 5G private network perception evaluation method provided for an embodiment of the present application;

[0057] Figure 7 A structural schematic diagram of a 5G private network perception evaluation device provided for an embodiment of the present application;

[0058] Figure 8 A structural schematic diagram of an electronic device provided for an embodiment of the present application.

[0059] Through the above-mentioned drawings, the specific embodiments of the present application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0060] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same or similar components. The embodiments described in the following exemplary embodiments are not meant to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0061] The terms "first", "second", "third", "fourth" and the like in the description and the claims of the present application and above-mentioned drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a specific sequential or chronological order. It is to be understood that the use of the terms so construed can be interchanged, such that, for example, embodiments of the present application described herein could operate in other sequences than those illustrated or described herein. Further, the terms "comprise" and "include" and variations thereof, as well as the terms "contain" and "comprise", are intended to cover a non-exclusive inclusion, such that a process, method, system, product or apparatus that comprises a list of steps or units not necessarily limited to those steps or units clearly recited, but can include other steps or units not expressly listed or inherent to such process, method, product or apparatus.

[0062] 5G network user perception evaluation plays a great role in how to better serve customers. In the prior art, the general application scenario is identified by manual method, and then the public user is analyzed for perception evaluation. Since the general application scenario closely related to the public user is relatively single and clear, there is no substantial problem in identifying the application scenario by manual method which affects the evaluation result.

[0063] However, with the development and application of many new network fields such as Internet, Internet of Things, Internet of Vehicles, intelligent manufacturing, wireless medical treatment, smart city and the like, 5G private network is applied to various new applications and new scene layers, showing the characteristics of differentiated business demand and diversified business scenarios. In the face of so many new business scenarios, it is not suitable to identify the application scenario by manual method. Not only is the dependence on personnel experience too great and the requirement too high, but also the identification speed for new application scenarios is too slow and the efficiency is too low, which is difficult to adapt to the rapid development of 5G business. Moreover, the key indicators of the current 5G private network mainly refer to 5G public customers, and there is a lack of quality perception analysis of upper business establishment and interaction scenarios, which is likely to cause excessive protection or service failure. Therefore, there is a problem of how to realize automatic and accurate identification of 5G private network application scenarios while improving the 5G private network perception evaluation result to better serve customers.

[0064] In view of the above technical problems, the present application provides a 5G private network perception evaluation method, device, electronic equipment and readable storage medium. The method proposes an application scenario classification logic, abstracts the manual identification of the application scenario into a mathematical model, deeply integrates the customer application scenario, constructs a perception evaluation model including a network support layer, a general performance layer and a business scenario layer, and calculates the perception evaluation comprehensive score in combination with the weight parameter and the threshold. The application scenario can be automatically identified, the applicability is higher, and the perception evaluation result can better reflect the customer business perception situation.

[0065] The technical solutions of the present application and how the technical solutions solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0066] Figure 1 The flowchart of the 5G private network perception evaluation method provided by an embodiment of the present application is shown. The execution subject of the embodiment can be a server or an electronic device with server function, such as a server shown in Figure 1 The 5G private network perception evaluation method provided by the embodiment includes:

[0067] S101, collecting real-time signaling plane data and real-time user plane data of the business to be evaluated in a preset time period.

[0068] Specifically, real-time signaling plane data and real-time user plane data of the to-be-evaluated service in a preset time period are collected to provide data support for subsequent perception evaluation. The preset time period refers to a time period segment of the to-be-evaluated service that needs to be evaluated, which can be 24 hours, one week, one month, or one quarter, and the present application does not make any limitation.

[0069] If it is intended to determine the perception situation of the to-be-evaluated service in the last month, real-time signaling plane data and real-time user plane data of the to-be-evaluated service in the last month are collected; if it is intended to determine the current perception situation of the to-be-evaluated service, real-time signaling plane data and real-time user plane data of the current to-be-evaluated service are collected. The closer the preset time period is to the current time, the better the current perception situation of the to-be-evaluated service can be reflected. In addition, real-time signaling plane data and real-time user plane data in special time periods, such as peak periods and valley periods of service operation, can also be collected to determine the perception situation of the to-be-evaluated service in various situations, so as to provide better and more accurate services.

[0070] The present application does not make any limitation on how to collect real-time signaling plane data and real-time user plane data of the to-be-evaluated service in a preset time period.

[0071] In some embodiments, the signaling collection system has collected full amount of signaling plane and user plane interface data by deploying a DPI probe (English full name: Deep Packet Inspection), and the target data can be obtained by directly interfacing with the signaling collection system.

[0072] DPI is a deep packet-based detection technology, which can perform deep detection on different network application layer loads, and determine the legitimacy by detecting the payload of the message. The DPI device can detect and analyze the traffic and message content at the key points of the network, filter and control the detected traffic according to the predefined strategy, and complete the functions of fine identification of services on the link, analysis of service traffic flow direction, statistics of service traffic proportion, shaping of service proportion, and application layer denial of service attack, filtering of viruses and Trojans, and control of abuse of P2P.

[0073] Therefore, real-time signaling plane data and real-time user plane data of the to-be-evaluated service in a preset time period can be directly obtained from the signaling collection system in which a DPI probe is deployed.

[0074] S102, acquiring a first set of key indicators corresponding to a network support layer and a second set of key indicators corresponding to a general performance layer according to the real-time signaling plane data.

[0075] Among them, the signaling plane is responsible for transmitting control signaling, and the user plane is for transmitting actual data. In fact, the user plane and the signaling plane are divided according to the type of data. In a communication system, there is generally a distinction between the user plane and the signaling plane. The user plane is the real service data, such as voice data or packet service data. The control plane is the signaling, which is used to control the call flow establishment, maintenance and release.

[0076] For example, the user plane of the wireless network layer is voice coding or packet data, that is, the real user data; the signaling plane of the wireless network layer is the control signaling protocol such as RANAP, RNSAP and NBAP, which is used to control the call flow.

[0077] Signaling is a control signal needed to ensure the normal communication of the whole network in addition to the transmission of user information in a wireless communication system. Signaling is different from user information, which is directly transmitted by the sender to the receiver through the communication network, while signaling usually needs to be transmitted between different links of the communication network, such as base stations, mobile stations, mobile control switching centers, etc. Each link analyzes and processes and interacts to form a series of operations and controls, which ensures the effective and reliable transmission of user information. Therefore, signaling can be regarded as the control system of the whole communication network, and its performance determines the ability and quality of a communication network to provide services for users to a great extent.

[0078] Different service scenarios have different service requirements for 5G private networks, but they all have the same demands for large bandwidth, low latency, high efficiency, high security, etc., which are closely related to the signaling plane data. The user plane data of different service scenarios must be different.

[0079] Therefore, the collected real-time signaling plane data is decoded, analyzed and processed to form indicators that can clearly reflect the above service requirements, and key indicators are extracted according to the importance of the indicators.

[0080] Exemplarily, the signaling plane key indicators can be reflected as the security mode activation success rate, the maximum number of connected users, the Xn handover success rate, the RRC connection abnormal release rate, etc. The signaling plane key indicators of different service scenarios are different.

[0081] In some embodiments, the key indicators extracted from the signaling plane data can also be divided into a first set of key indicators of the network support layer and a second set of key indicators of the general performance layer for distinction.

[0082] The network support layer mainly solves the problems of node positioning and time positioning. For example, the data obtained in a sensor network must be combined with the corresponding location information to ensure the effectiveness of the information, so node positioning is an important problem to be solved.

[0083] Different services have different requirements for the 5G private network. In some embodiments, the network support layer can present network access, bearer establishment, service request, and location update processes in the service establishment process as key indicators to form a first key indicator set of the network support layer.

[0084] The general performance layer can start from the Open System Interconnection Communication Reference Model (OSI for short). The OSI model is a conceptual model proposed by the International Organization for Standardization, which is a standard framework that attempts to interconnect computers worldwide into a network. It can provide developers with a necessary and general concept to develop a perfect framework that can be used to explain the connection of different systems. The OSI divides the computer network architecture into seven layers, namely the physical layer, the data link layer, the network layer, the transport layer, the session layer, the presentation layer, and the application layer.

[0085] Each of them has its own different functions. The physical layer is used to convert data into electronic signals that can be transmitted through physical media, which is equivalent to a mail carrier in a post office. The data link layer determines the way to access the network medium. The network layer is used to route data through large networks, which is equivalent to a sorting worker in a post office. The transport layer is used to provide reliable connection between terminals, which is equivalent to a delivery clerk in a company. The session layer allows users to establish connections using simple and memorable names, which is equivalent to a secretary who receives, writes, and opens envelopes in a company. The presentation layer is used to negotiate data exchange formats, which is equivalent to an assistant who briefs the boss and writes letters for the boss. The application layer is the interface between the user's application program and the network.

[0086] In some embodiments, the general performance layer can present the general performance protocols of the network layer, the transport layer, and the application layer as key indicators to form a second key indicator set of the general performance layer.

[0087] It can be understood that the above description should be understood as a possible implementation manner, rather than a limitation thereof.

[0088] S103、According to the real-time user plane data and the preset application scenario feature library, a third key indicator set corresponding to the service scenario layer is determined.

[0089] Specifically, since different service scenarios have different network service requirements for the 5G private network, in order to make the perception evaluation result more accurate, in addition to evaluating the key indicators determined by the signaling plane data, it is also necessary to evaluate and analyze in combination with the specific service scenario.

[0090] The user plane represents real service data, and therefore the actual application scenario of the service to be evaluated can be determined according to real-time user plane data of the service to be evaluated, and then the third key indicator set of the corresponding service scenario layer is determined. Specifically, the real-time user plane data of the service to be evaluated can be decoded, analyzed and processed to extract service features of the service to be evaluated, and then the service features of the service to be evaluated are compared with the service features in the preset application scenario feature library, so that the third key indicator set corresponding to the service scenario layer of the service to be evaluated is determined.

[0091] Exemplarily, the user plane key indicators can include service success rate, service rate, service delay, service drop rate, etc., and different service scenarios generally have different user plane key indicators.

[0092] In the present application, the manner of comparing the service features of the service to be evaluated with the service features in the preset application scenario feature library is not limited.

[0093] In some embodiments, the similarity can be calculated by a certain clustering algorithm, and the third key indicator set corresponding to the service scenario layer of the service to be evaluated is determined by comparing the similarity. If the similarity is greater than a preset threshold, the third key indicator set corresponding to the service features in the preset application scenario feature library is taken as the third key indicator set corresponding to the service scenario layer of the service to be evaluated; if the similarity is less than the preset threshold, a new third key indicator set is established according to the service to be evaluated.

[0094] S104, constructing a 3-layer perception evaluation model according to the first key indicator set, the second key indicator set and the third key indicator set.

[0095] Specifically, according to the first key indicator set corresponding to the network support layer, the second key indicator set corresponding to the general performance layer, and the third key indicator set corresponding to the service scenario layer determined above, the perception evaluation process can be abstracted into a 3-layer perception evaluation model including the network support layer, the general performance layer and the service scenario layer.

[0096] Since different types of application scenarios have different perception requirements, on the basis of the 3-layer perception evaluation model, a weight configuration module and a threshold setting module with flexible settings can also be designed to adapt to different application scenarios.

[0097] Exemplarily, Figure 2 A structural schematic diagram of a perception evaluation model provided by the present application is shown in FIG. 1. As shown in FIG. 1, the perception evaluation model includes a network support layer, a general performance layer and a service scenario layer. Figure 2As shown, the perception evaluation model includes three layers of network support layer, general performance layer, and service scenario layer. The first set of key indicators corresponding to the network support layer includes network access, bearer establishment, service request, and location update. The second set of key indicators corresponding to the general performance layer includes network layer, transmission layer, and application layer. The service scenario layer includes one scenario, and the third set of key indicators corresponding to the scenario includes indicators n1, n2, and so on. When a same customer has multiple application scenarios, the service scenario layer can have multiple scenarios.

[0098] S105, scoring according to the perception evaluation model and the preset threshold, and calculating a perception evaluation comprehensive score in combination with the weight parameter.

[0099] Since different types of application scenarios have different perception requirements for the 5G private network, different weight parameters and preset thresholds can be flexibly configured for different application scenarios when performing perception evaluation analysis on the 5G private network, so as to more accurately obtain the perception evaluation comprehensive score.

[0100] The comprehensive scoring method is a quantitative processing method for projects divided into grades according to quality through scoring, and can be used for comprehensive evaluation of qualitative ordering problems. The core content is to assign different scores to different grades of evaluation, and to perform comprehensive evaluation based on this.

[0101] Optionally, in some embodiments, the perception evaluation comprehensive score is calculated in combination with the weight parameter according to scoring according to the perception evaluation model and the preset threshold, including:

[0102] The network support layer, the general performance layer, the service scenario layer, and the corresponding first set of key indicators, the second set of key indicators, and the third set of key indicators are configured with different weight parameters and preset thresholds. Then, the perception evaluation comprehensive score is calculated in combination with the weight parameter according to scoring according to the perception evaluation model and the preset threshold.

[0103] Specifically, each key indicator in the network support layer, the general performance layer, the service scenario layer, and the corresponding first set of key indicators, the second set of key indicators, and the third set of key indicators in the perception evaluation model of the to-be-evaluated service is configured with different weight parameters and preset thresholds. Then, each key indicator of the to-be-evaluated service determined above is compared with the preset threshold, and each key indicator is scored. Then, the perception evaluation comprehensive score is calculated according to the scoring results and the corresponding weight parameters.

[0104] In this application, how to configure specific weight parameters and preset thresholds is not limited. In some embodiments, the weight parameters and the recommended values of the preset thresholds can be directly configured according to expert experience, or can be calculated in combination with certain AI algorithms.

[0105] The 5G private network perception evaluation method provided by the embodiment of the application comprises the following steps.

[0106] Figure 3 The flowchart of the 5G private network perception evaluation method provided by another embodiment of the application is shown in the figure. Figure 3 The 5G private network perception evaluation method provided by the embodiment of the application comprises the following steps.

[0107] S301, real-time signaling plane data and real-time user plane data of a to-be-evaluated service in a preset time period are collected.

[0108] S302, a first key indicator set corresponding to a network support layer and a second key indicator set corresponding to a general performance layer are acquired according to the real-time signaling plane data.

[0109] S303, a third key indicator set corresponding to a service scenario layer is determined according to the real-time user plane data and a preset application scenario feature library.

[0110] The specific implementation of steps S301, S302 and S303 is similar to that of steps S101, S102 and S103, and will not be described here.

[0111] Optionally, in some embodiments, before the third key indicator set corresponding to the service scenario layer is determined according to the real-time user plane data and the preset application scenario feature library, the establishment of the preset application scenario feature library is further included.

[0112] The establishment of the preset application scenario feature library comprises:

[0113] S3001, collect full amount of signaling offline data and user plane offline data as sample data.

[0114] Specifically, the signaling offline data and user plane offline data of the 5G private network users are collected as sample data to provide massive data support for subsequent training. The signaling offline data and user plane offline data can be obtained by connecting the signaling collection system in which the DPI probe is deployed.

[0115] S3002, perform clustering operation on the sample data through unsupervised learning, and output a feature vector.

[0116] The unsupervised learning is used to solve various problems in pattern recognition according to the unlabeled training sample data. In the big data algorithm, the clustering algorithm is generally used as the basis for analysis of other algorithms, and the data is clustered to analyze some characteristics of the data as a whole. There are many clustering algorithms, and k-means is the simplest and most practical algorithm. The most commonly used similarity measurement methods in k-means algorithm include Euclidean distance method, cosine similarity function, Manhattan distance, etc., and different methods are usually used in different situations. The specific implementation of unsupervised learning clustering operation is not limited in the present application.

[0117] Specifically, the unlabeled user plane offline data obtained is decoded, analyzed, associated and processed, and then the clustering operation of unsupervised learning is performed, the data is divided into multiple clusters through the similarity of the data, and the similarity within the cluster is as large as possible, while the similarity between the clusters is as small as possible. After the unsupervised learning clustering is initially completed, the application scenario feature vector is output synchronously, that is, the corresponding relationship between the cluster and the feature vector is formed.

[0118] After a certain feature vector set is formed, the sample data can also be labeled by using the known feature vector set, and the accuracy and efficiency of clustering can be improved by using the supervised learning clustering method.

[0119] Supervised learning is a machine learning method that infers a function from a labeled training data set. Supervised learning uses samples with known characteristics as a training set to establish a mathematical model, such as the discriminant model in pattern recognition, the weight model in artificial neural network method, etc., and then uses the established model to predict unknown samples.

[0120] Exemplarily, Figure 4 An application scenario feature recognition diagram provided by the present application. As shown in Figure 4As shown, the sample data is sequentially subjected to data processing and unsupervised clustering operation to output the feature vectors of each cluster, and then the model calibration is performed through supervised learning to improve the accuracy and efficiency of clustering.

[0121] S3003, a mapping relationship of the application scenario, the feature vector, and the third key indicator set is constructed, and a preset application scenario feature library is output.

[0122] Specifically, after the application scenario feature recognition of the user plane offline data is completed, the corresponding relationship of the cluster and the feature vector has been obtained, and then the feature selection and processing of the feature vector according to its importance can output the corresponding key indicator set, and then the mapping relationship of the application scenario, the feature vector, and the key indicator set can be constructed to output the preset application scenario feature library.

[0123] Optionally, in some embodiments, constructing the mapping relationship of the application scenario, the feature vector, and the third key indicator set comprises:

[0124] annotating the feature vector and dividing it into the corresponding application scenario; establishing the third key indicator set for each application scenario to construct the mapping relationship of the application scenario, the feature vector, and the third key indicator set.

[0125] Specifically, after the above steps S3001 and S3002, the corresponding relationship of the cluster and the feature vector is obtained, then the feature vector is annotated by experts, and each cluster is divided into the corresponding application scenario, and then the corresponding third key indicator set is established for each application scenario through expert experience, so as to form the mapping relationship of the application scenario, the feature vector, and the third key indicator set, and output the application scenario feature library containing the known application scenario, the corresponding feature vector, and the corresponding third key indicator set as the preset application scenario feature library.

[0126] Exemplarily, Figure 5 A schematic diagram of a preset application scenario feature library provided by the present application is shown. As Figure 5 shown, the preset application scenario feature library contains n application scenarios, each application scenario has a corresponding feature vector and a key indicator set.

[0127] On the basis of the above establishment of the preset application scenario feature library, in some embodiments, step S303 determines the third key indicator set corresponding to the business scenario layer according to the real-time user plane data and the preset application scenario feature library, comprising:

[0128] S3031, the real-time user plane data is formatted to form a real-time feature vector;

[0129] S3032, similarity calculation is performed between the real-time feature vector and the feature vector in the preset application scenario feature library;

[0130] S3033, determine the application scenario corresponding to the real-time user plane data to output the third key indicator set corresponding to the service scenario layer.

[0131] Specifically, after collecting the real-time user plane data of the to-be-evaluated service in a preset time period, the real-time user plane data is decoded, analyzed, associated, etc. to generate a text file, and the key fields of the real-time user plane data such as user number, location information, etc. are associated and filled back. And according to the feature dimensions of the preset application scenario feature library, the real-time user plane data is formatted and arranged to form the real-time feature vector of the to-be-evaluated service.

[0132] Then, the real-time feature vector and the feature vector in the preset application scenario feature library are calculated for similarity, and according to the size of the similarity, the application scenario corresponding to the real-time user plane data and the key indicator set corresponding to the application scenario can be determined, that is, the third key indicator set corresponding to the service scenario layer of the to-be-evaluated service can be output.

[0133] Among them, when calculating the similarity between the real-time feature vector and the feature vector in the preset application scenario feature library, Chebyshev distance or Euclidean distance can be used for calculation, and the smaller the distance between the two, the greater the similarity between the two.

[0134] Optionally, if the similarity is greater than or equal to a preset threshold, the application scenario corresponding to the real-time user plane data is determined according to the preset application scenario feature library;

[0135] If the similarity is less than the preset threshold, the application scenario is recommended according to the preset algorithm, and the real-time feature vector is labeled to supplement to the preset application scenario feature library.

[0136] Specifically, if the similarity is greater than or equal to a preset threshold, it indicates that there is a similar application scenario to the application scenario of the to-be-evaluated service in the preset application scenario feature library, and the application scenario corresponding to the real-time user plane data can be determined according to the preset application scenario feature library to output the third key indicator set corresponding to the service scenario layer corresponding to the to-be-evaluated service.

[0137] If the similarity is less than the preset threshold, it indicates that there is no similar application scenario to the application scenario of the to-be-evaluated service in the preset application scenario feature library, and the to-be-evaluated service is a new application scenario. At this time, the closest application scenario can be provided according to the preset algorithm to output the third key indicator set corresponding to the service scenario layer corresponding to the to-be-evaluated service. At the same time, the real-time feature vector is labeled to form a mapping relationship of a new application scenario, a feature vector, and a key indicator set, and is supplemented to the preset application scenario feature library to enrich the preset application scenario feature library.

[0138] The size of the preset threshold is not limited, and can be continuously adjusted according to the establishment of the preset application scene feature library.

[0139] S304, constructing a three-layer perception evaluation model according to the first key indicator set, the second key indicator set, and the third key indicator set.

[0140] S305, scoring according to the perception evaluation model and the preset threshold, and calculating a perception evaluation comprehensive score in combination with a weight parameter.

[0141] The specific implementation of steps S304 and S305 is similar to that of steps S104 and S105, and will not be described here.

[0142] Optionally, in some embodiments, after step S305, scoring according to the perception evaluation model and the preset threshold, and calculating a perception evaluation comprehensive score in combination with a weight parameter, the method further includes:

[0143] S306, presenting a perception evaluation result and outputting a quality defect indicator, the perception evaluation result being determined according to the perception evaluation comprehensive score, and the quality defect indicator being an indicator that needs to be optimized.

[0144] Specifically, after calculating the perception evaluation comprehensive score, the perception evaluation result is presented, and the quality defect indicator is outputted. The perception evaluation result is a specific embodiment of the perception evaluation comprehensive score, and the quality defect indicator can be an indicator in the key indicator set that does not meet the standard, which needs to be optimized.

[0145] For example, the score of an indicator that does not meet the preset threshold is relatively low, which will affect the result of the perception evaluation comprehensive score and needs to be optimized. Therefore, the indicator is outputted to remind and guide the network optimization personnel of the 5G private network to optimize, so as to provide better services for customers.

[0146] For example, Figure 6 A flowchart of a 5G private network perception evaluation method provided by an embodiment of the present application is shown in FIG. 1. Figure 6 As shown in FIG. 1, the whole process of 5G private network perception evaluation can be divided into two parts: offline analysis and online analysis. In offline analysis, application scene feature library is established through offline business data collection and application scene feature recognition; in online analysis, application scene classification is performed through real-time business data acquisition, business feature extraction, and similarity calculation with the features in the application scene feature library, then a perception evaluation model is constructed, calculation is performed in combination with weight configuration, and finally perception indicators are presented. The specific implementation of the above content has been described in detail in the above embodiments, and will not be described here.

[0147] The 5G private network perception evaluation method provided in the embodiment of the application is based on the previous embodiment, full-quantity collected signaling plane offline data and user plane offline data are used as sample data, and a feature vector is output through data processing and clustering operation, a mapping relationship of an application scenario, a feature vector and a key index set is constructed to output a preset application scenario feature library, similarity calculation is performed on real-time user plane data and the preset application scenario feature library to determine a third key index set corresponding to a business scenario. The identification of the application scenario is abstracted into a mathematical model, and the application scenario does not need to be identified by artificial means, and is no longer dependent on experts, and the identification is faster and more efficient. In addition, after obtaining the perception evaluation comprehensive score, the evaluation result is displayed, and an index that has a greater impact on the evaluation result and needs to be optimized is output, so that network optimization personnel can more intuitively determine the content that needs to be optimized, and better service can be provided for customers.

[0148] The following is an apparatus embodiment of the application, which can be used to execute the method embodiments of the application. For details not disclosed in the apparatus embodiments of the application, refer to the method embodiments of the application.

[0149] Figure 7 The structure diagram of the 5G private network perception evaluation device provided in the embodiment of the application is shown in FIG. 7. Figure 7 As shown in FIG. 7, the 5G private network perception evaluation device 70 includes an acquisition module 701, a processing module 702, a model establishment module 703 and an evaluation module 704.

[0150] The acquisition module 701 is configured to collect real-time signaling plane data and real-time user plane data of a business to be evaluated in a preset time period.

[0151] The processing module 702 is configured to acquire a first key index set corresponding to a network support layer and a second key index set corresponding to a general performance layer according to the real-time signaling plane data, and determine a third key index set corresponding to a business scenario layer according to the real-time user plane data and a preset application scenario feature library.

[0152] The model establishment module 703 is configured to construct a three-layer perception evaluation model according to the first key index set, the second key index set and the third key index set.

[0153] The evaluation module 704 is configured to calculate a perception evaluation comprehensive score according to the perception evaluation model and a preset threshold score in combination with a weight parameter.

[0154] Optionally, the 5G private network perception evaluation device further includes a training module.

[0155] The training module is configured to collect full-volume signaling offline data and user plane offline data as sample data before determining the third key indicator set corresponding to the service scenario layer according to real-time user plane data and a preset application scenario feature library; perform clustering operation on the sample data through unsupervised learning to output a feature vector; and construct a mapping relationship among the application scenario, the feature vector, and the third key indicator set, and output the preset application scenario feature library.

[0156] Optionally, the training module is further configured to label the feature vector and divide the feature vector into a corresponding application scenario; establish the third key indicator set for each application scenario to construct the mapping relationship among the application scenario, the feature vector, and the third key indicator set.

[0157] Optionally, the processing module 702 is further configured to perform format processing on the real-time user plane data to form a real-time feature vector; perform similarity calculation on the real-time feature vector and the feature vector in the preset application scenario feature library; determine the application scenario corresponding to the real-time user plane data to output the third key indicator set corresponding to the service scenario layer.

[0158] Optionally, the processing module 702 is further configured to, if the similarity is greater than or equal to a preset threshold, determine the application scenario corresponding to the real-time user plane data according to the preset application scenario feature library; and if the similarity is less than the preset threshold, recommend the application scenario according to a preset algorithm and label the real-time feature vector to supplement to the preset application scenario feature library.

[0159] Optionally, the evaluation module 704 is further configured to configure different weight parameters and preset thresholds for the network support layer, the general performance layer, the service scenario layer, and the corresponding first key indicator set, the second key indicator set, and the third key indicator set; score according to the perception evaluation model and the preset threshold, and calculate a perception evaluation comprehensive score in combination with the weight parameters.

[0160] Optionally, the 5G private network perception evaluation apparatus further includes a presentation module.

[0161] The presentation module is configured to present the perception evaluation result and output a quality difference indicator, the perception evaluation result being determined according to the perception evaluation comprehensive score, and the quality difference indicator being an indicator that needs to be optimized.

[0162] The 5G private network perception evaluation apparatus provided in this embodiment can be used to execute the 5G private network perception evaluation method of any one of the above-mentioned embodiments, and has similar implementation principles and technical effects, which will not be described herein again.

[0163] It should be noted that the division of the various modules of the above apparatus is only a logical functional division, and in actual implementation, all or part of them can be integrated into one physical entity, or can be physically separated. These modules can all be implemented in the form of software invoked by a processing element; all can be implemented in the form of hardware; or some modules can be implemented in the form of software invoked by a processing element, and some modules can be implemented in the form of hardware. For example, the object detection module can be a separately established processing element, or can be integrated into a chip of the above apparatus. In addition, it can also be stored in the form of program code in the memory of the above apparatus, and the function of the above data processing module is invoked and executed by a processing element of the above apparatus. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or can be independently implemented. The processing element here can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of the hardware in the processor element or the instructions in the form of software.

[0164] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), etc. For another example, when a certain module above is implemented in the form of program code invoked by a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can invoke program code. For another example, these modules can be integrated together to implement in the form of a system on a chip (SOC).

[0165] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available media sets. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)) and the like.

[0166] Figure 8 The structure schematic diagram of the electronic device provided by the embodiments of the present application is shown in the figure. As shown in the figure, the electronic device 80 includes a processor 801 and a memory 802 in communication with the processor. Figure 8

[0167] The memory 802 stores computer execution instructions; the processor 801 executes the computer execution instructions stored in the memory 802 to realize any one of the 5G private network perception evaluation methods as described above.

[0168] In the specific implementation of the above electronic device, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC) and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor for execution.

[0169] ​The embodiment of the present application further provides a computer readable storage medium, which stores computer execution instructions. The computer execution instructions are executed by a processor to implement the 5G private network sensing evaluation method according to any one of the preceding embodiments.

[0170] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by computer instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various storage medium capable of storing program codes, such as ROM, RAM, magnetic disc or optical disc.

[0171] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The specification and examples given are exemplary only and the true scope and spirit of the application are indicated by the following claims. It will be appreciated by those skilled in the art that you can make various modifications and improvements without departing from the scope of the application. The scope of the application is indicated by the following claims.

[0172] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is indicated by the appended claims.

Claims

1. A 5G private network perception evaluation method, characterized in that, include: Collect real-time signaling plane data and real-time user plane data of the service to be evaluated within a preset time period; Based on the real-time signaling plane data, obtain the first set of key indicators corresponding to the network support layer and the second set of key indicators corresponding to the general performance layer; The third set of key indicators corresponding to the business scenario layer is determined based on the real-time user plane data and the preset application scenario feature library. A three-layer perception evaluation model is constructed based on the first set of key indicators, the second set of key indicators, and the third set of key indicators. Based on the perception evaluation model and the preset threshold score, the comprehensive perception evaluation score is calculated in combination with the weight parameters. Before determining the third key indicator set corresponding to the business scenario layer based on the real-time user plane data and the preset application scenario feature library, the method further includes: Collect all offline signaling plane data and offline user plane data to use as sample data; The sample data is clustered using unsupervised learning to output a feature vector. The feature vectors are labeled and assigned to corresponding application scenarios; For each application scenario, a third key indicator set is established to construct a mapping relationship between the application scenario, the feature vector, and the third key indicator set, and the preset application scenario feature library is output.

2. The 5G private network perception evaluation method according to claim 1, characterized in that, The step of determining the third key indicator set corresponding to the business scenario layer based on the real-time user plane data and the preset application scenario feature library includes: The real-time user plane data is formatted to form a real-time feature vector; The similarity calculation is performed between the real-time feature vector and the feature vector in the preset application scenario feature library; Determine the application scenario corresponding to the real-time user plane data to output the third key indicator set corresponding to the business scenario layer.

3. The 5G private network perception evaluation method according to claim 2, characterized in that, The process of determining the application scenario corresponding to the real-time user plane data includes: If the similarity is greater than or equal to a preset threshold, the application scenario corresponding to the real-time user plane data is determined according to a preset application scenario feature library. If the similarity is less than a preset threshold, then an application scenario is recommended according to a preset algorithm, and the real-time feature vector is labeled to supplement the preset application scenario feature library.

4. The 5G private network perception evaluation method according to any one of claims 1-3, characterized in that, The step of calculating a comprehensive perception evaluation score based on the perception evaluation model and preset threshold scoring, combined with weight parameters, includes: Different weight parameters and preset thresholds are configured for the network support layer, the general performance layer, the business scenario layer, and the corresponding first key indicator set, second key indicator set, and third key indicator set; The comprehensive score of the perception evaluation is calculated based on the perception evaluation model and the preset threshold score, combined with the weight parameters.

5. The 5G private network perception evaluation method according to claim 4, characterized in that, The method further includes: The system presents the perception evaluation results and outputs the quality poor index. The perception evaluation results are determined based on the comprehensive perception evaluation score, and the quality poor index is the index that needs to be optimized.

6. A 5G private network sensing and evaluation device, characterized in that, include: The acquisition module is used to collect real-time signaling plane data and real-time user plane data of the service to be evaluated within a preset time period; The processing module is used to obtain the first key indicator set corresponding to the network support layer and the second key indicator set corresponding to the general performance layer based on the real-time signaling plane data. And determine the third key indicator set corresponding to the business scenario layer based on the real-time user plane data and the preset application scenario feature library; The model building module is used to construct a three-layer perception evaluation model based on the first key indicator set, the second key indicator set, and the third key indicator set. The evaluation module is used to calculate the comprehensive evaluation score based on the perception evaluation model and the preset threshold score, combined with the weight parameters. The device further includes: a training module; The training module is specifically used for: Collect all offline signaling plane data and offline user plane data to use as sample data; The sample data is clustered using unsupervised learning to output a feature vector. The feature vectors are labeled and assigned to corresponding application scenarios; For each application scenario, a third key indicator set is established to construct a mapping relationship between the application scenario, the feature vector, and the third key indicator set, and the preset application scenario feature library is output.

7. An electronic device, comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-5.

Images