Communication method and device

Abstract

The invention discloses a communication method and device, relates to the field of communication, and can automatically realize availability verification of a model, sample set construction, execution of a model training / fine tuning task and the like in an intelligent scene. In the method, a data analysis network element establishes a secure channel with an automatic dial testing system at an end side through a first device, tasks such as dial testing, data acquisition and labeling, flow statistics and the like can be issued to the automatic dial testing system, and dial testing terminal equipment is driven to trigger corresponding application service access. The method supports the execution of data cleaning and labeling tasks at the end side, and then reports the related labeling data to the data analysis network element through the first device, thereby enabling the data analysis network element to carry out the comparison with an intelligent service perception and experience evaluation system in the network based on the labeling data. And automatic verification, sample collection and model fine-tuning closed-loop capabilities of an intelligent scene are realized.

Description

Technical Field

[0001] This application relates to the field of communications, and more particularly to communication methods and apparatus. Background Technology

[0002] In intelligent scenarios defined by the 3rd Generation Partnership Project (3GPP), such as service awareness (SA) scenarios for application identification and quality of experience (QoE) scenarios for business experience evaluation, the usability of these intelligent scenarios needs to be assessed through deployed artificial intelligence (AI) / machine learning (ML) models. However, for intelligent scenarios, obtaining labeled data samples for model training and validation requires a combination of human effort and tools to perform tasks such as probing, packet capture, data analysis and annotation, network data collection, and result comparison to improve model accuracy. Therefore, in intelligent scenarios, how to automatically verify model usability (e.g., whether model inference results show deviations), and how to automate application (APP) probing, packet capture and data annotation, sample acquisition, and training tasks when the usability of AI models deployed on the live network shows deviations, are urgent research topics. Summary of the Invention

[0003] This application provides a communication method and apparatus that establishes a channel for authentication, control commands, and annotation data exchange between an end-side automatic probing system and an NWDAF deployed in the network. This enables automated verification of model availability, construction of sample sets, and execution of model training / fine-tuning tasks in intelligent scenarios.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] Firstly, a communication method is provided. This method can be executed by a data analysis network element, or by a component of the data analysis network element, such as a processor, chip, or chip system of the data analysis network element, or by a logic module or software capable of implementing all or part of the data analysis network element. The method is applied to a data analysis network element, and when a secure channel is established with a first automatic dialing test system via a first device, it includes: sending a first task request to a user plane network element, the first task request requesting the user plane network element to acquire and report first data of the service flow accessing a first application by the first automatic dialing test system; sending a second task request to the first device, the second task request requesting the first device to trigger the first automatic dialing test system to acquire and report labeled data of the service flow accessing the first application; wherein the first data and labeled data are used to complete a first task related to service inference or training of a first model, and the first model is used for service-related inference; receiving the first data from the user plane network element and receiving the labeled data from the first device.

[0006] In this method, after establishing a secure channel with the first automatic dialing test system on the edge through the first device, the data analysis network element sends task requests related to the first task of inference or training of the first model to the user plane network element and the first device respectively. This triggers the first automatic dialing test system on the edge to perform dialing tests on the first application to obtain the labeled data reported by the first automatic dialing test system. It also triggers the user plane network element to collect the service flow data of the dialing test access to the first application from the user plane to obtain the first data reported by the user plane network element. The first task is completed based on the labeled data and the first data. For example, if the first data is the service flow access data obtained by the user plane network element from the statistics of the collected service flows, the data analysis network element can verify the inference results of the first model by comparing the labeled data and the first data. Or, if the first data is the service flow feature data extracted by the user plane network element from the service flows accessed by the automatic dialing test system, the data analysis network element associates the labeled data with the first data to generate a sample set for training the first model. Therefore, the data analysis network element can build an end-to-end automatic testing, collection, analysis, and labeling framework through the secure channel established by the first device. After deployment in the existing intelligent network scenario, it can automatically realize model availability verification, on-network collection of labeled sample sets, on-network model training and fine-tuning tasks to improve model capabilities.

[0007] In one possible design, the first automatic testing system includes a first terminal device for testing a first application. By testing the application on the first terminal device, usability verification, sample collection, and model training or fine-tuning related to the specified application can be performed.

[0008] In one possible design, the first device triggering the first automatic testing system to acquire and report labeled data of the service flow accessing the first application may include: the first device triggering the first automatic testing system to initiate testing of the first application, packet capture of testing data, cleaning and traffic statistics of testing data, collection and labeling of testing data, reporting of labeled data, and the first device initiating monitoring and reporting of the labeled data reported by the first automatic testing system. Thus, the data analysis network element can decompose the first task, allocating tasks to the first device and user plane network elements, and can automatically realize model availability verification, sample set construction, and model training / fine-tuning tasks in intelligent scenarios.

[0009] In one possible design, the first task could be to verify whether the business reasoning results of the first model meet expectations, or the first task could be to generate a sample set for training the first model. Therefore, the first task could be a task related to model reasoning verification or model training in an intelligent scenario.

[0010] In one possible design, if the first task is to verify whether the business inference result of the first model meets expectations, then the first data is used for comparison with the labeled data. The first data may include the business inference result, business flow access data obtained by the user plane network element from the collected business flow of the first application, and 5-tuple information. The business inference result is obtained by the user plane network element using the first model to infer the business flow access data obtained by the user plane network element. Therefore, when the first task is related to verifying the model inference result, the first data includes the business inference result obtained by the user plane network element based on the first model, which is used to compare with the actual result in the labeled data reported by the endpoint to determine whether the inference result of the first model meets expectations. In other words, the first data is business flow access data used to verify whether the business inference result of the first model meets expectations by comparing it with the labeled data. For example, if the business inference result is the application type of the accessed business flow, the business flow access data may include the accessed business flow traffic, total traffic, etc.

[0011] In one possible design, if the first task is to generate a sample set for training the first model, then the first data, used to associate with the labeled data, may include business flow feature data and quintuple information of the business flow collected by the user plane network elements from the first application. Thus, since the first task is related to model training, the first data includes sample feature data extracted by the user plane network elements from the collected business flow to train the model's business inference capabilities. This sample data is then used to form sample data with the labeled data reported from the edge. In other words, the first data is business flow feature data used to construct training samples with the labeled data. For example, to train the first model's business quality assessment capability, the business flow feature data may include business flow statistical feature information, business flow load feature information, and protocol layer header fields of the business flow, among other information or data.

[0012] In one possible design, the business inference result can be a business identification result or a business quality assessment result, and the sample set is used to train the business identification capability or business quality assessment capability of the first model. Therefore, the first task can be an intelligent business perception task or an intelligent business quality assessment task in an intelligent scenario.

[0013] In one possible design, if the business inference result is a business identification result or the sample set is used to train the business identification capability of the first model, then the labeled data may include application identifiers as labels and business flow access data and quintuple information associated with the application identifiers. If the business inference result is a business quality assessment result or the sample set is used to train the business quality assessment capability of the first model, then the labeled data may include business quality results as labels and business flow access data and quintuple information associated with the business quality results. Therefore, the data type used as labels in the labeled data varies depending on the task type. For example, business flow access data associated with business quality results may include key quality indicators (KQIs) related to evaluating business quality results (such as video mean opinion score (vMOS)). KQIs include resolution, bitrate, first frame loading time, first frame loading time target achievement rate, stuttering time, number of stutters, video playback success rate, startup download rate, media data transmission round-trip latency, media data transmission packet loss rate, and download rate.

[0014] In one possible design, establishing a secure channel between the data analysis network element and the first automatic dialing testing system via the first device can include: after completing security authentication of the first automatic dialing testing system via the first device, establishing a secure channel between the data analysis network element and the first automatic dialing testing system. This secure channel supports the data analysis network element in issuing tasks to the first automatic dialing testing system and in reporting labeled data to the first automatic dialing testing system. Therefore, the establishment of a secure channel between the data analysis network element and the first automatic dialing testing system via the first device is achieved through the first device completing security authentication of the first automatic dialing testing system.

[0015] In one possible design, the first device can be a user plane network element. The security authentication of the first automatic dialing test system can be performed through the first device, which may include: sending a first authentication request to the user plane network element. The first authentication request requests the user plane network element to perform security authentication of the automatic dialing test system to be monitored. The first authentication request includes the first authentication information of the automatic dialing test system to be monitored, and the automatic dialing test system to be monitored includes the first automatic dialing test system. Therefore, the data analysis network element can perform security authentication of the first automatic dialing test system through the user plane network element to establish a secure channel with the first automatic dialing test system through the user plane network element.

[0016] In one possible design, the first authentication information may include the authentication information of the terminal device used for testing in the automatic dialing test system to be monitored. The authentication information of the terminal device includes the identifier of the terminal device, the authentication account of the terminal device, and the authentication key of the terminal device.

[0017] In one possible design, the first device can be a dial-up testing server corresponding to the first automatic dial-up testing system. The security authentication of the first automatic dial-up testing system via the first device can include: receiving a second authentication request from the dial-up testing server, the second authentication request requesting the data analysis network element to perform security authentication on the dial-up testing server, the second authentication request including the authentication information of the dial-up testing server; and sending a second authentication response to the dial-up testing server, the second authentication response notifying the dial-up testing server of successful security authentication. Thus, after the data analysis network element performs security authentication on the independently configured dial-up testing server, it can perform security authentication on the first automatic dial-up testing system through the dial-up testing server, thereby establishing a secure channel between the two systems.

[0018] In one possible design, the authentication information of the test server includes the address of the test server, the authentication account of the test server, and the authentication key of the test server.

[0019] In one possible design, the method described in the first aspect may further include: receiving a first authentication response from a first device, the first authentication response being used to notify the data analysis network element that the first automatic dialing test system has successfully authenticated. Thus, the data analysis network element can determine whether the first automatic dialing test system has successfully authenticated based on the feedback from the first device.

[0020] In one possible design, the first device can be a security bastion host. The first automatic dialing test system also includes an automatic dialing test control device for controlling the dialing test of a first terminal device of a first application. The security authentication of the first automatic dialing test system via the first device can include: receiving a third authentication request from the security bastion host, the third authentication request being used to request the data analysis network element to perform security authentication on the first automatic dialing test system, the third authentication request including the authentication information of the first automatic dialing test system; and sending a third authentication response to the security bastion host, the third authentication response being used to notify the first automatic dialing test system of successful security authentication. Thus, the data analysis network element can perform security authentication on the first automatic dialing test system through the security bastion host to establish a secure channel with the first automatic dialing test system.

[0021] Secondly, a communication method is provided. This method can be executed by a first device, or by a component of the first device, such as a processor, chip, or chip system of the first device, or by a logic module or software capable of implementing all or part of the first device. The method is applied to the first device, and when the first device completes the establishment of a secure channel between a data analysis network element and a first automatic dialing testing system, it includes: receiving a second task request from the data analysis network element, the second task request being used to request the first device to trigger the first automatic dialing testing system to acquire and report labeled data of the service flow accessing the first application; sending a third task request to the first automatic dialing testing system according to the second task request, the third task request being used to request the first automatic dialing testing system to acquire and report labeled data of the service flow accessing the first application; receiving labeled data from the first automatic dialing testing system, the labeled data being used to complete a first task related to service inference or training of a first model, the first model being used for service-related inference; and sending the labeled data to the data analysis network element.

[0022] In this method, the first device can establish a secure channel between the data analysis network element and the first automatic testing system on the edge. Based on the task request issued by the data analysis network element, the first automatic testing system is triggered to perform application testing to obtain labeled data. The labeled data is then sent to the data analysis network element to complete the first task related to the inference or training of the first model. Thus, the first device supports the construction of a secure channel between the automatic testing system and the data analysis network element, and enables functions such as business access information statistics and reporting, sample feature collection, and labeled data acquisition. This makes automated verification, sample collection, and closed-loop fine-tuning of AI models in intelligent scenarios possible.

[0023] In one possible design, the first automatic dialing test system includes a first terminal device for dialing test of a first application; sending a third task request to the first automatic dialing test system according to a second task request may include: sending a third task request to the first terminal device according to the second task request; receiving annotation data from the first automatic dialing test system may include: receiving annotation data from the first terminal device.

[0024] In one possible design, the first device triggering the first automatic dialing test system to acquire and report the labeled data of the service flow accessing the first application may include: the first device triggering the first automatic dialing test system to start dialing test on the first application, dialing test data packet capture, dialing test data cleaning and traffic statistics, dialing test data collection and labeling, labeled data reporting, and the first device starting to monitor and report the labeled data reported by the first automatic dialing test system.

[0025] In one possible design, the first task could be to verify whether the business reasoning results of the first model meet expectations, or the first task could be to generate a sample set for training the first model.

[0026] In one possible design, the business reasoning result can be either a business identification result or a business quality assessment result, and the sample set can be used to train the business identification capability or business quality assessment capability of the first model.

[0027] In one possible design, if the business inference result is a business identification result or the sample set is used to train the business identification capability of the first model, the labeled data may include application identifiers as labels and business flow access data and quintuple information associated with the application identifiers; if the business inference result is a business quality assessment result or the sample set is used to train the business quality assessment capability of the first model, the labeled data may include business quality results as labels and business flow access data and quintuple information associated with the business quality results.

[0028] In one possible design, the first device is a user plane network element, and the method described in the second aspect may further include: receiving a first task request from a data analysis network element, the first task request being used to request the user plane network element to acquire and report first data of the service flow of the first automatic dialing system accessing the first application, the first data being used to complete the first task; and sending the first data to the data analysis network element.

[0029] In one possible design, if the first task is to verify whether the business inference result of the first model meets expectations, then the first data is used to compare with the labeled data. The first data may include the business inference result, the business flow access data obtained by the user plane network element from the collected business flow of the first application, and the quintuple information. The business inference result is obtained by the user plane network element using the first model to infer the business flow data obtained by the user plane network element.

[0030] In one possible design, if the first task is to generate a sample set for training the first model, then the first data is used to associate with the labeled data. The first data may include service flow feature data and quintuple information obtained by the user plane network element from the service flow of the first application.

[0031] In one possible design, the first device establishes a secure channel between the data analysis network element and the first automatic dial-up testing system. This can include: receiving a first authentication request from the data analysis network element, the first authentication request requesting the user plane network element to perform security authentication on the automatic dial-up testing system to be monitored, the first authentication request including first authentication information of the automatic dial-up testing system to be monitored, and the automatic dial-up testing system to be monitored including the first automatic dial-up testing system; receiving a fourth authentication request from the first automatic dial-up testing system, the fourth authentication request requesting security authentication on the first automatic dial-up testing system, the fourth authentication request including authentication information of the first automatic dial-up testing system; and, based on the first authentication information and the authentication information of the first automatic dial-up testing system, sending a first authentication response to the data analysis network element and a fourth authentication response to the first automatic dial-up testing system, the first authentication response notifying the data analysis network element that the first automatic dial-up testing system has successfully completed security authentication, and the fourth authentication response notifying the first automatic dial-up testing system that the security authentication has been successfully completed.

[0032] In one possible design, the first authentication information may include the authentication information of the terminal device used for testing in the automatic dialing test system to be monitored. The authentication information of the terminal device includes the identifier of the terminal device, the authentication account of the terminal device, and the authentication key of the terminal device.

[0033] In one possible design, the first device can be a dial-up testing server, corresponding to a first automatic dial-up testing system. The first device establishes a secure channel between the data analysis network element and the first automatic dial-up testing system, which may include: sending a second authentication request to the data analysis network element, the second authentication request requesting the data analysis network element to perform security authentication on the dial-up testing server, the second authentication request including the authentication information of the dial-up testing server; receiving a second authentication response from the data analysis network element, the second authentication response notifying the dial-up testing server of successful security authentication; receiving a fourth authentication request from the first automatic dial-up testing system, the fourth authentication request requesting security authentication on the first automatic dial-up testing system, the fourth authentication request including the authentication information of the first automatic dial-up testing system; and sending a fourth authentication response to the first automatic dial-up testing system based on the locally configured authentication information of the automatic dial-up testing system to be monitored and the authentication information of the first automatic dial-up testing system, the fourth authentication response notifying the first automatic dial-up testing system of successful security authentication.

[0034] In one possible design, the first device can be a security bastion host, and the first automatic dialing test system includes an automatic dialing test control device for controlling the dialing test of a first terminal device of a first application. The first device establishes a secure channel between the data analysis network element and the first automatic dialing test system, and may include: receiving a fourth authentication request from the automatic dialing test control device, the fourth authentication request requesting security authentication of the first automatic dialing test system, the fourth authentication request including the authentication information of the first automatic dialing test system, including the address of the automatic dialing test control device; sending a third authentication request to the data analysis network element based on the authentication information of the first automatic dialing test system, the third authentication request requesting the data analysis network element to perform security authentication of the first automatic dialing test system, the third authentication request including the authentication information of the first automatic dialing test system; receiving a third authentication response from the data analysis network element, the third authentication response notifying the first automatic dialing test system of successful security authentication; and sending a fourth authentication response to the first automatic dialing test system, the fourth authentication response notifying the first automatic dialing test system of successful security authentication.

[0035] The description of the technical effects of the method described in the second aspect can be found in the relevant description of the technical effects of the method described in the first aspect above, and will not be repeated here.

[0036] Thirdly, a communication method is provided. This method can be executed by a first automatic dialing test system, or by a component of the first automatic dialing test system, such as its processor, chip, or chip system. It can also be implemented by a logic module or software capable of implementing all or part of the first automatic dialing test system. This method is applied to the first automatic dialing test system. When a secure channel is established between the system and a data analysis network element via a first device, the method includes: receiving a third task request from the first device, the third task request requesting the first automatic dialing test system to acquire and report labeled data of the service flow accessing the first application; and sending labeled data to the first device, the labeled data being used to complete a first task related to service inference or training of a first model, the first model being used for service-related inference.

[0037] In one possible design, the first automatic dialing test system includes a first terminal device for dialing test of a first application; receiving a third task request from a first device may include: the first terminal device receiving the third task request from the first device; and sending annotation data to the first device may include: the first terminal device sending annotation data to the first device.

[0038] In one possible design, the first automatic testing system acquiring and reporting labeled data of the business flow accessing the first application may include: the first automatic testing system initiating testing of the first application, packet capture of testing data, cleaning and traffic statistics of testing data, collection and labeling of testing data, and reporting of labeled data.

[0039] In one possible design, the first task is to verify whether the business reasoning results of the first model meet expectations, or the first task is to generate a sample set for training the first model.

[0040] In one possible design, the business reasoning result is either a business identification result or a business quality assessment result, and the sample set is used to train the business identification capability or business quality assessment capability of the first model.

[0041] In one possible design, if the business inference result is a business identification result or the sample set is used to train the business identification capability of the first model, the labeled data may include application identifiers as labels and business flow access data and quintuple information associated with the application identifiers; if the business inference result is a business quality assessment result or the sample set is used to train the business quality assessment capability of the first model, the labeled data may include business quality results as labels and business flow access data and quintuple information associated with the business quality results.

[0042] In one possible design, establishing a secure channel between the first device and the data analysis network element may include: sending a fourth authentication request to the first device, the fourth authentication request requesting security authentication of the first automatic dialing test system, the fourth authentication request including authentication information of the first automatic dialing test system; and receiving a fourth authentication response from the first device, the fourth authentication response notifying the first automatic dialing test system that the security authentication was successful.

[0043] In one possible design, the first automatic dialing test system includes a first terminal device for dialing test of a first application. The authentication information of the first automatic dialing test system includes the address of the dialing test server and the authentication information of the first terminal device. The authentication information of the first terminal device includes the identifier of the first terminal device, the authentication account of the first terminal device, and the authentication key of the first terminal device. The dialing test server corresponds to the first automatic dialing test system.

[0044] In one possible design, the first device can be a user plane network element or a dial-up test server.

[0045] In one possible design, the first device can be a security bastion host, and the first automatic dialing test system includes an automatic dialing test control device for controlling the dialing test of a first terminal device of the first application. The authentication information of the first automatic dialing test system also includes the address of the automatic dialing test control device.

[0046] In one possible design, sending a fourth authentication request to the first device may include: the automatic dialing control device sending a fourth authentication request to the first device; receiving a fourth authentication response from the first device may include: the automatic dialing control device receiving a fourth authentication response from the first device.

[0047] The description of the technical effects of the method described in the third aspect can be found in the relevant description of the technical effects of the method described in the first aspect above, and will not be repeated here.

[0048] Fourthly, a communication device is provided for implementing the various methods described above. This communication device can be a data analysis network element as described in the first aspect, or a device containing the aforementioned data analysis network element, or a device included in the aforementioned data analysis network element, such as a chip. The communication device includes corresponding modules, units, or means for implementing the methods described in the first aspect. These modules, units, or means can be implemented in hardware, software, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the aforementioned functions.

[0049] In some possible designs, where the communication device establishes a secure channel with the first automatic dialing test system via the first device, the communication device includes a processing module and a transceiver module. The transceiver module is used to send a first task request to a user plane network element, requesting the user plane network element to acquire and report first data of the service flow accessing the first application by the first automatic dialing test system. The transceiver module is also used to send a second task request to the first device, requesting the first device to trigger the first automatic dialing test system to acquire and report labeled data of the service flow accessing the first application; wherein the first data and labeled data are used to complete a first task related to service inference or training of a first model, and the first model is used for service-related inference. The transceiver module is also used to receive the first data from the user plane network element and the labeled data from the first device. The processing module is used to complete the first task based on the first data and labeled data.

[0050] In one possible design, the first automatic dialing test system includes a first terminal device for dialing test of a first application.

[0051] In one possible design, the first device triggering the first automatic dialing test system to acquire and report the labeled data of the service flow accessing the first application may include: the first device triggering the first automatic dialing test system to start dialing test on the first application, dialing test data packet capture, dialing test data cleaning and traffic statistics, dialing test data collection and labeling, labeled data reporting, and the first device starting to monitor and report the labeled data reported by the first automatic dialing test system.

[0052] In one possible design, the first task could be to verify whether the business reasoning results of the first model meet expectations, or the first task could be to generate a sample set for training the first model.

[0053] In one possible design, if the first task is to verify whether the business inference result of the first model meets expectations, then the first data is used to compare with the labeled data. The first data may include the business inference result, the business flow access data obtained by the user plane network element from the collected business flow of the first application, and the quintuple information. The business inference result is obtained by the user plane network element using the first model to infer the business flow access data obtained by the user plane network element.

[0054] In one possible design, if the first task is to generate a sample set for training the first model, then the first data is used to associate with the labeled data. The first data may include business flow feature data and quintuple information of the business flow of the first application collected by the user plane network element.

[0055] In one possible design, the business reasoning result can be either a business identification result or a business quality assessment result, and the sample set is used to train the business identification capability or business quality assessment capability of the first model.

[0056] In one possible design, if the business inference result is a business identification result or the sample set is used to train the business identification capability of the first model, the labeled data may include application identifiers as labels and business flow access data and quintuple information associated with the application identifiers; if the business inference result is a business quality assessment result or the sample set is used to train the business quality assessment capability of the first model, the labeled data may include business quality results as labels and business flow access data and quintuple information associated with the business quality results.

[0057] In one possible design, establishing a secure channel between the first device and the first automatic dialing test system may include: after the first device completes the security authentication of the first automatic dialing test system, a transceiver module is used to establish a secure channel between the first device and the first automatic dialing test system. The secure channel supports the communication device to send tasks to the first automatic dialing test system and supports the first automatic dialing test system to report labeled data.

[0058] In one possible design, the first device can be a user plane network element. The first device completes the security authentication of the first automatic dialing test system. It can include: a transceiver module, used to send a first authentication request to the user plane network element. The first authentication request is used to request the user plane network element to perform security authentication on the automatic dialing test system to be monitored. The first authentication request includes the first authentication information of the automatic dialing test system to be monitored. The automatic dialing test system to be monitored includes the first automatic dialing test system.

[0059] In one possible design, the first authentication information may include the authentication information of the terminal device used for testing in the automatic dialing test system to be monitored. The authentication information of the terminal device includes the identifier of the terminal device, the authentication account of the terminal device, and the authentication key of the terminal device.

[0060] In one possible design, the first device can be a dial-up testing server corresponding to the first automatic dial-up testing system. The first device performs security authentication of the first automatic dial-up testing system and may include: a transceiver module for receiving a second authentication request from the dial-up testing server. The second authentication request requests the communication device to perform security authentication on the dial-up testing server and includes the authentication information of the dial-up testing server. The transceiver module is also used to send a second authentication response to the dial-up testing server, which notifies the dial-up testing server that the security authentication was successful.

[0061] In one possible design, the authentication information of the test server includes the address of the test server, the authentication account of the test server, and the authentication key of the test server.

[0062] In one possible design, the transceiver module is also used to receive a first authentication response from the first device, which is used to notify the communication device that the first automatic dialing test system has successfully authenticated the security.

[0063] In one possible design, the first device can be a security bastion host. The first automatic dialing test system further includes an automatic dialing test control device for controlling the dialing test of a first terminal device of the first application. The first device performs security authentication of the first automatic dialing test system and may include: a transceiver module for receiving a third authentication request from the security bastion host. The third authentication request requests the communication device to perform security authentication on the first automatic dialing test system and includes authentication information of the first automatic dialing test system. The transceiver module is also used to send a third authentication response to the security bastion host, which notifies the first automatic dialing test system that security authentication was successful.

[0064] In one possible design, the transceiver module may include a receiving module and a sending module. The sending module implements the sending function of the communication device described in the fourth aspect, and the receiving module implements the receiving function of the communication device described in the fourth aspect.

[0065] In one possible design, the communication device described in the fourth aspect may further include a storage module storing programs or instructions. When the processing module executes the program or instructions, the communication device described in the fourth aspect can perform the method described in the first aspect.

[0066] Fifthly, a communication device is provided for implementing the various methods described above. This communication device may be the first device described in the second aspect, or a device comprising the first device, or a device included in the first device, such as a chip. The communication device includes corresponding modules, units, or means for implementing the methods described in the second aspect. These modules, units, or means may be implemented in hardware, software, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the functions described above.

[0067] In some possible designs, after the communication device establishes a secure channel between the data analysis network element and the first automatic testing system, the communication device includes a processing module and a transceiver module. The transceiver module receives a second task request from the data analysis network element, requesting the communication device to trigger the first automatic testing system to acquire and report labeled data of the service flow accessing the first application. The processing module controls the transceiver module to send a third task request to the first automatic testing system based on the second task request, requesting the first automatic testing system to acquire and report labeled data of the service flow accessing the first application. The transceiver module also receives labeled data from the first automatic testing system, the labeled data being used to complete a first task related to service inference or training of the first model, the first model being used for service-related inference. The transceiver module also sends labeled data to the data analysis network element.

[0068] In one possible design, the first automatic dialing test system includes a first terminal device for dialing test of a first application; a processing module for controlling the transceiver module to send a third task request to the first automatic dialing test system according to a second task request, which may include: the processing module for controlling the transceiver module to send the third task request to the first terminal device according to the second task request; and receiving annotation data from the first automatic dialing test system, which may include: receiving annotation data from the first terminal device.

[0069] In one possible design, the communication device triggering the first automatic dialing test system to acquire and report the labeled data of the service flow accessing the first application may include: the communication device triggering the first automatic dialing test system to start dialing test on the first application, dialing test data packet capture, dialing test data cleaning and traffic statistics, dialing test data collection and labeling, labeled data reporting, and the communication device starting to monitor and report the labeled data reported by the first automatic dialing test system.

[0070] In one possible design, the first task could be to verify whether the business reasoning results of the first model meet expectations, or the first task could be to generate a sample set for training the first model.

[0071] In one possible design, the business reasoning result can be either a business identification result or a business quality assessment result, and the sample set can be used to train the business identification capability or business quality assessment capability of the first model.

[0072] In one possible design, if the business inference result is a business identification result or the sample set is used to train the business identification capability of the first model, the labeled data may include application identifiers as labels and business flow access data and quintuple information associated with the application identifiers; if the business inference result is a business quality assessment result or the sample set is used to train the business quality assessment capability of the first model, the labeled data may include business quality results as labels and business flow access data and quintuple information associated with the business quality results.

[0073] In one possible design, the communication device is a user plane network element. The method described in the second aspect may further include: receiving a first task request from a data analysis network element, the first task request being used to request the user plane network element to acquire and report first data of the service flow of the first automatic dialing system accessing the first application, the first data being used to complete the first task; and sending the first data to the data analysis network element.

[0074] In one possible design, if the first task is to verify whether the business inference result of the first model meets expectations, then the first data is used to compare with the labeled data. The first data may include the business inference result, the business flow access data obtained by the user plane network element from the collected business flow of the first application, and the quintuple information. The business inference result is obtained by the user plane network element using the first model to infer the business flow access data obtained by the user plane network element.

[0075] In one possible design, if the first task is to generate a sample set for training the first model, then the first data is used to associate with the labeled data. The first data may include service flow feature data and quintuple information obtained by the user plane network element from the service flow of the first application.

[0076] In one possible design, the communication device establishes a secure channel between the data analysis network element and the first automatic dial-up testing system. This can include: a transceiver module for receiving a first authentication request from the data analysis network element. The first authentication request requests the user plane network element to perform security authentication on the automatic dial-up testing system to be monitored. The first authentication request includes first authentication information of the automatic dial-up testing system to be monitored, and the automatic dial-up testing system to be monitored includes the first automatic dial-up testing system. The transceiver module is also used to receive a fourth authentication request from the first automatic dial-up testing system. The fourth authentication request requests security authentication on the first automatic dial-up testing system and includes authentication information of the first automatic dial-up testing system. The transceiver module is further used to send a first authentication response to the data analysis network element and a fourth authentication response to the first automatic dial-up testing system based on the first authentication information and the authentication information of the first automatic dial-up testing system. The first authentication response notifies the data analysis network element that the first automatic dial-up testing system has successfully completed security authentication, and the fourth authentication response notifies the first automatic dial-up testing system that the security authentication has been successful.

[0077] In one possible design, the first authentication information may include the authentication information of the terminal device used for testing in the automatic dialing test system to be monitored. The authentication information of the terminal device includes the identifier of the terminal device, the authentication account of the terminal device, and the authentication key of the terminal device.

[0078] In one possible design, the communication device can be a dial-up testing server, corresponding to a first automatic dial-up testing system. The communication device establishes a secure channel between the data analysis network element and the first automatic dial-up testing system. It can include: a transceiver module for sending a second authentication request to the data analysis network element, requesting the data analysis network element to perform security authentication on the dial-up testing server, and the second authentication request includes the authentication information of the dial-up testing server. The transceiver module is also used to receive a second authentication response from the data analysis network element, notifying the dial-up testing server of successful security authentication. The transceiver module is further used to receive a fourth authentication request from the first automatic dial-up testing system, requesting security authentication on the first automatic dial-up testing system, and the fourth authentication request includes the authentication information of the first automatic dial-up testing system. The transceiver module is also used to send a fourth authentication response to the first automatic dial-up testing system based on the locally configured authentication information of the automatic dial-up testing system to be monitored and the authentication information of the first automatic dial-up testing system, the fourth authentication response notifying the first automatic dial-up testing system of successful security authentication.

[0079] In one possible design, the communication device can be a security bastion host. The first automatic dialing test system includes an automatic dialing test control device for controlling the dialing test of a first terminal device of a first application. The communication device establishes a secure channel between the data analysis network element and the first automatic dialing test system and may include: a transceiver module for receiving a fourth authentication request from the automatic dialing test control device. The fourth authentication request requests security authentication of the first automatic dialing test system and includes authentication information of the first automatic dialing test system, including the address of the automatic dialing test control device. The transceiver module is also used to send a third authentication request to the data analysis network element based on the authentication information of the first automatic dialing test system. The third authentication request requests the data analysis network element to perform security authentication of the first automatic dialing test system and includes authentication information of the first automatic dialing test system. The transceiver module is also used to receive a third authentication response from the data analysis network element, which notifies the first automatic dialing test system of successful security authentication. The transceiver module is also used to send a fourth authentication response to the first automatic dialing test system, which notifies the first automatic dialing test system of successful security authentication.

[0080] In one possible design, the transceiver module may include a receiving module and a sending module. The sending module implements the sending function of the communication device described in the fifth aspect, and the receiving module implements the receiving function of the communication device described in the fifth aspect.

[0081] In one possible design, the communication device described in the fifth aspect may further include a storage module storing programs or instructions. When the processing module executes the program or instructions, the communication device described in the fifth aspect can perform the method described in the second aspect.

[0082] Sixthly, a communication device is provided for implementing the various methods described above. This communication device may be the first automatic dialing system described in the third aspect, or a device comprising the first automatic dialing system, or a device included in the first automatic dialing system, such as a chip. The communication device includes corresponding modules, units, or means for implementing the methods described in the three aspects above. These modules, units, or means may be implemented in hardware, software, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the functions described above.

[0083] In some possible designs, where a secure channel is established between the communication device and the data analysis network element via the first device, the communication device includes a processing module and a transceiver module. The transceiver module receives a third task request from the first device, requesting the communication device to acquire and report labeled data of the service flow accessing the first application. The processing module generates the labeled data. The transceiver module also sends the labeled data to the first device, the labeled data being used to complete a first task related to service inference or training of the first model, the first model being used for service-related inference.

[0084] In one possible design, the communication device includes a first terminal device for dialing a first application; a transceiver module for receiving a third task request from the first device, which may include: the first terminal device receiving the third task request from the first device; the transceiver module is also used to send annotation data to the first device, which may include: the first terminal device sending annotation data to the first device.

[0085] In one possible design, the communication device acquiring and reporting labeled data of the service flow accessing the first application may include: the communication device initiating a dial test on the first application, dial test data packet capture, dial test data cleaning and traffic statistics, dial test data collection and labeling, and labeled data reporting.

[0086] In one possible design, the first task is to verify whether the business reasoning results of the first model meet expectations, or the first task is to generate a sample set for training the first model.

[0087] In one possible design, the business reasoning result is either a business identification result or a business quality assessment result, and the sample set is used to train the business identification capability or business quality assessment capability of the first model.

[0088] In one possible design, if the business inference result is a business identification result or the sample set is used to train the business identification capability of the first model, the labeled data may include application identifiers as labels and business flow access data and quintuple information associated with the application identifiers; if the business inference result is a business quality assessment result or the sample set is used to train the business quality assessment capability of the first model, the labeled data may include business quality results as labels and business flow access data and quintuple information associated with the business quality results.

[0089] In one possible design, establishing a secure channel between the first device and the data analysis network element may include: a transceiver module, further configured to send a fourth authentication request to the first device, the fourth authentication request requesting security authentication of the communication device, the fourth authentication request including authentication information of the communication device. The transceiver module is also configured to receive a fourth authentication response from the first device, the fourth authentication response notifying the communication device of successful security authentication.

[0090] In one possible design, the communication device includes a first terminal device for testing a first application. The authentication information of the communication device includes the address of the testing server and the authentication information of the first terminal device. The authentication information of the first terminal device includes the identifier of the first terminal device, the authentication account of the first terminal device, and the authentication key of the first terminal device. The testing server corresponds to the communication device.

[0091] In one possible design, the first device can be a user plane network element or a dial-up test server.

[0092] In one possible design, the first device can be a security bastion host, and the communication device includes an automatic dialing control device for controlling the dialing of a first terminal device of the first application. The authentication information of the communication device also includes the address of the automatic dialing control device.

[0093] In one possible design, the transceiver module is further configured to send a fourth authentication request to the first device, which may include: the automatic dialing control device sending the fourth authentication request to the first device; the transceiver module is further configured to receive a fourth authentication response from the first device, which may include: the automatic dialing control device receiving the fourth authentication response from the first device.

[0094] In one possible design, the transceiver module may include a receiving module and a transmitting module. The transmitting module implements the transmitting function of the communication device described in the sixth aspect, and the receiving module implements the receiving function of the communication device described in the sixth aspect.

[0095] In one possible design, the communication device described in the sixth aspect may further include a storage module storing programs or instructions. When the processing module executes the program or instructions, the communication device described in the sixth aspect can perform the method described in the third aspect.

[0096] A seventh aspect provides a communication device (e.g., the communication device may be a chip or a chip system). The communication device includes a processor for implementing the functions involved in any of the preceding aspects.

[0097] In one possible design, the communication device may further include a memory for storing necessary program instructions and data. A processor is coupled to the memory and is used to execute the computer program or instructions stored in the memory, causing the communication device to perform the method described in any of the possible implementations of the first to third aspects.

[0098] In one possible design, the communication device described in the seventh aspect may further include a transceiver. This transceiver may be a transceiver circuit or an interface circuit. The transceiver can be used for communication between the communication device described in the seventh aspect and other communication devices.

[0099] In one possible design, the processor can be integrated with the memory.

[0100] In some possible designs, when the device is a chip system, it can be composed of chips or contain chips and other discrete components.

[0101] Eighthly, a communication device is provided, the communication device including a processor and an interface circuit, the interface circuit being configured to receive signals from other communication devices outside the communication device and transmit them to the processor or to send signals from the processor to other communication devices outside the communication device, the processor being configured to implement the method as described in any one of the possible implementations of the first to third aspects via logic circuits or execution code instructions.

[0102] It is understood that when the communication device provided by either the seventh or eighth aspect is a chip, the aforementioned transmitting action / function can be understood as an output, and the aforementioned receiving action / function can be understood as an input.

[0103] Ninth aspect, a computer-readable storage medium is provided that stores a computer program or instructions that, when executed on a communication device, enable the communication device to perform the method described in any one of the first to third aspects.

[0104] In a tenth aspect, a computer program product comprising instructions is provided, including computer program code, which, when executed on a communication device, enables the communication device to perform the method described in any one of the first to third aspects.

[0105] Eleventhly, a communication system is provided, comprising: a data analysis network element for implementing the method described in the first aspect, a first device for implementing the method described in the second aspect, and a first automatic dialing system for implementing the method described in the third aspect.

[0106] In a twelfth aspect, a communication chip is provided, wherein instructions are stored that, when the chip is operated on a communication device, cause the method described in any one of the first to third aspects above to be implemented. Attached Figure Description

[0107] Figure 1 This is a schematic diagram of a network architecture applicable to embodiments of this application;

[0108] Figure 2 This is a schematic diagram of an open architecture for network data analysis.

[0109] Figure 3 This is a schematic diagram of the online fine-tuning process for models in intelligent scenarios;

[0110] Figure 4 This application provides a schematic diagram of the architecture of a communication system.

[0111] Figures 5-8 A flowchart illustrating a communication method provided in an embodiment of this application;

[0112] Figure 9 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0113] Figure 10 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application. Detailed Implementation

[0114] To better understand the embodiments of this application, the following points are explained before introducing the embodiments of this application.

[0115] First, in the embodiments of this application, "for indicating" can include both direct and indirect indication. When describing a certain "indication information" for indicating A, it can include whether the indication information directly indicates A or indirectly indicates A, but does not necessarily mean that the indication information carries A.

[0116] The information indicated by the instruction is called the information to be instructed. In the specific implementation process, there are many ways to indicate the information to be instructed, such as, but not limited to, directly indicating the information to be instructed, such as the information to be instructed itself or its index. It can also be indirectly indicated by indicating other information, where there is a relationship between the other information and the information to be instructed. It can also indicate only a part of the information to be indicated, while the other parts are known or pre-agreed upon. For example, the instruction of specific information can be achieved by using a pre-agreed (e.g., protocol-defined) arrangement of various pieces of information, thereby reducing instruction overhead to some extent. At the same time, common parts of various pieces of information can be identified and indicated uniformly to reduce the instruction overhead caused by individually indicating the same information.

[0117] Furthermore, the specific indication method can also be any existing indication method, such as, but not limited to, the above-mentioned indication methods and their various combinations. Specific details of various indication methods can be found in existing technologies, and will not be repeated here. As described above, for example, when multiple pieces of information of the same type need to be indicated, the indication methods for different pieces of information may differ. In the specific implementation process, the required indication method can be selected according to specific needs. This application embodiment does not limit the selected indication method; therefore, the indication methods involved in this application embodiment should be understood to cover various methods that enable the party to be indicated to obtain the information to be indicated.

[0118] The information to be instructed can be sent as a whole or divided into multiple sub-information messages, and the sending period and / or timing of these sub-information messages can be the same or different. This application does not limit the specific sending method. The sending period and / or timing of these sub-information messages can be predefined, for example, according to a protocol, or configured by the transmitting device by sending configuration information to the receiving device.

[0119] Second, in the embodiments of this application, the terms "first," "second," and various numerical designations are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. For example, the first authentication request and the second authentication request are only used to distinguish different authentication messages and do not limit their order. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that the terms "first," "second," etc., are not necessarily different.

[0120] Third, in the embodiments of this application, descriptions such as "when," "under the circumstances," "if," and "if" all refer to the fact that the device (e.g., the first device) will make corresponding processing under certain objective circumstances. They are not time limits, nor do they require the device (e.g., the first device) to have a judgment action when it is implemented, nor do they mean that there are other limitations.

[0121] In this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or explanation. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0122] In this application, "sending information" can be understood as one device sending information to another device, or it can also be understood as one logical module within a device sending information to another logical module. For example, "access network device sending information" can be understood as the access network device sending information to another device (such as a terminal device), or it can be understood as logical module 1 in the access network device sending information to logical module 2 in the access network device.

[0123] In this application, "receiving information" can be understood as one device receiving information from another device, or it can also be understood as a logical module within a device receiving information from another logical module. For example, "access network device receiving information" can be understood as the access network device receiving information from another device (such as a terminal device), or it can be understood as logical module 1 in the access network device receiving information from logical module 2 in the access network device.

[0124] In this application, phrases such as "sending information to... (e.g., a terminal device)" or related illustrations in the accompanying drawings can be understood as the destination of the information being a terminal. This can include sending information directly or indirectly to a terminal. Similarly, phrases such as "receiving information from... (e.g., a terminal device)," "receiving information from... (e.g., a terminal device)," or "receiving information sent (e.g., by a terminal device)," or related illustrations in the accompanying drawings, can be understood as the source of the information being a terminal. This can include receiving information directly or indirectly from a terminal. Information may undergo necessary processing between the source and destination, such as format changes, but the destination can understand the valid information from the source. Similar expressions in this application can be interpreted similarly and will not be elaborated further here.

[0125] Finally, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0126] This application will present various aspects, embodiments, or features relating to a system that may include multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all the devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches may also be used.

[0127] The technical solutions of this application embodiment can be applied to various communication systems, such as the worldwide interoperability for microwave access (WiMAX) communication system, the 4th generation (4G) mobile communication system, such as the long term evolution (LTE) system, the 5th generation (5G) mobile communication system, such as the new radio (NR) system, and future communication systems.

[0128] In communication systems, the portion operated by the operator can be referred to as a public land mobile network (PLMN), or operator network, etc. A PLMN is a network established and operated by the government or its approved operators for the purpose of providing terrestrial mobile communication services to the public. It is primarily a public network where mobile network operators (MNOs) provide mobile broadband access services to users. The PLMN described in this application embodiment can specifically be a network conforming to 3GPP standards, or simply a 3GPP network. 3GPP networks typically include, but are not limited to, 5G networks, 4G networks, and other future communication systems.

[0129] See Figure 1 , Figure 1 This is a schematic diagram of a network architecture applicable to embodiments of this application, taking a 5G network architecture based on service-based architecture (SBA) in a non-roaming scenario as defined in the 3GPP standardization process as an example. Figure 1As shown, this network architecture may include terminal devices, an access network (AN), and a core network (CN). Terminal devices access the data network (DN) through access network devices and core network devices.

[0130] The aforementioned terminal equipment can be a terminal device with transceiver functions, or a chip or chip system that can be installed in the terminal device. This terminal equipment can also be referred to as user equipment (UE), access terminal, subscriber unit, user station, mobile station (MS), mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device. The terminal devices in the embodiments of this application may be mobile phones, cellular phones, smartphones, tablets, wireless data cards, personal digital assistants (PDAs), wireless modems, handsets, laptop computers, machine-type communication (MTC) terminals, computers with wireless transceiver capabilities, virtual reality (VR) terminals, augmented reality (AR) terminals, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, vehicle-mounted terminals, roadside units (RSUs) with terminal functions, etc. The terminal device of this application may also be an on-board module, on-board unit, on-board component, on-board chip or on-board unit that is built into a vehicle as one or more components or units.

[0131] (R)AN is used to implement access-related functions, providing network access for authorized users in specific areas and determining transmission links of different qualities based on user level and service requirements to transmit user data. (R)AN forwards control signals and user data between the terminal device and the CN. (R)AN may include one or more access network devices, also referred to as access network nodes, radio access network (R)AN nodes, (R)AN entities, or access nodes, etc., used to help terminal devices achieve wireless access, and is a device with wireless transceiver capabilities or a chip or chip system that can be installed in the device. The interface between the access network device and the terminal device can be a Uu interface (or air interface, i.e., messages exchanged between the access network device and the terminal device can be called air interface messages). Of course, in future communications, the interface name may remain unchanged or be replaced by other names; this application does not limit this.

[0132] The access network equipment includes, but is not limited to: base stations, evolved NodeBs (eNodeB / eNBs), access points (APs), transmission reception points (TRPs or transmission points (TPs), next-generation NodeBs (gNBs), base stations in future mobile communication systems, or access nodes in wireless fidelity (Wi-Fi) systems. Access network equipment can be macro base stations, micro base stations or indoor stations, relay nodes or donor nodes, open radio access networks (O-RAN), or radio controllers in centralized radio access network (C-RAN) scenarios. Access network equipment can also be one or a group of antenna panels (including multiple antenna panels) of a 5G base station, or it can be a network node constituting a gNB, TRP, TP, or transmission measurement function (TMF), such as a central unit (CU), a distributed unit (DU), a CU-control plane (CP), a CU-user plane (UP), or a radio unit (RU), or an RSU with base station functions. Optionally, access network equipment can also be a server, wearable device, vehicle, or in-vehicle equipment. For example, the access network equipment in vehicle-to-everything (V2X) technology can be an RSU. All or part of the functions of the access network equipment in this application can also be implemented through software functions running on hardware, or through virtualization functions instantiated on a platform (e.g., a cloud platform). The access network equipment in this application can also be a logical node, logical module, or software capable of implementing all or part of the access network equipment functions.

[0133] In this network, CU and DU can be configured separately or included in the same network element, such as a baseband unit (BBU). RU can be included in radio frequency equipment or radio frequency units, such as remote radio units (RRUs), active antenna units (AAUs), or remote radio heads (RRHs). It is understood that access network equipment can be CU nodes, DU nodes, or equipment including both CU and DU nodes. Furthermore, CUs can be classified as network equipment within the access network RAN ​​or as network equipment within the CN; no restrictions are placed here.

[0134] In different systems, CU (or CU-CP and CU-UP), DU, or RU may have different names, but those skilled in the art will understand their meaning. For example, in an O-RAN system, CU can also be called O-CU (Open CU), DU can also be called O-DU, CU-CP can also be called O-CU-CP, CU-UP can also be called O-CU-UP, and RU can also be called O-RU. For ease of description, this application uses CU, CU-CP, CU-UP, DU, and RU as examples. Any of the units among CU (or CU-CP, CU-UP), DU, and RU in this application can be implemented through software modules, hardware modules, or a combination of software and hardware modules.

[0135] Access network equipment and terminal equipment can be fixed or mobile. They can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; they can also be deployed on water; and they can be deployed in the air on aircraft, balloons, and satellites. The embodiments of this application do not limit the application scenarios of the access network equipment and terminal equipment.

[0136] A CN may include, but is not limited to, the following network functions (NFs): userplane function (UPF), authentication server function (AUSF), access and mobility management function (AMF), session management function (SMF), network slice selection function (NSSF), network exposure function (NEF), network repository function (NRF), policy control function (PCF), unified data management (UDM), unified data repository (UDR), network data analytics function (NWDAF), and application function (AF).

[0137] The following is a brief explanation of the NF functions included in CN.

[0138] 1. UPF: Primarily responsible for user data processing (forwarding, receiving, billing, etc.). For example, a UPF can receive user data from a DN and forward it to the terminal device through the access network equipment. A UPF can also receive user data from the terminal device through the access network equipment and forward it to the DN. In a Protocol Data Unit (PDU) session, the UPF directly connected to the DN via N6 is also called the Protocol Data Unit Session Anchor (PSA).

[0139] 2. AUSF: Primarily used for performing security authentication of terminal devices.

[0140] 3. AMF: Primarily used for mobility management in mobile networks. Examples include user location updates, user network registration, and user handover.

[0141] 4. SMF: Primarily used for session management in mobile networks. This includes session establishment, modification, and release. Specific functions include assigning Internet Protocol (IP) addresses to users and selecting a UPF (User-Defined Provider) to provide packet forwarding capabilities.

[0142] 5. NSSF: Primarily used to select network slices for terminal devices.

[0143] 6. NEF: Primarily used to support the opening of capabilities and events. For example, NEF can expose some capabilities of the 5G network to third-party applications through application programming interfaces (APIs). Third-party applications can obtain some capabilities of the 5G network by calling the APIs provided by NEF through AF, enabling them to control certain behaviors of the 5G network and terminal devices.

[0144] 7. NRF: Primarily used to provide network element discovery functionality. Based on requests from other network elements, it provides network element information corresponding to the network element type. It also provides network element management services, such as network element registration, updating, deregistration, and network element status subscription and push.

[0145] 8. PCF: Primarily supports providing a unified policy framework to control network behavior, delivering policy rules to control layer network functions, and acquiring user subscription information related to policy decisions. PCF can provide policies to AMF and SMF, such as Quality of Service (QoS) policies and slice selection policies.

[0146] 9. UDM: Primarily used to store user data, such as contract data, authentication / authorization data, etc.

[0147] 10. UDR: Primarily used to store structured data, including contract data, policy data, externally exposed structured data, and application-related data.

[0148] 11. NWDAF: Primarily used to collect data (including data from terminal devices, access network devices, core network elements, and third-party application devices, or one or more of these). This data can be data from the terminal device, access network device, core network element, or third-party application device itself, or data from the terminal device on that access network device, core network element, or third-party application device. The collected data is then analyzed, and the analysis results are output for use by the network, network management equipment, and applications in decision-making. NWDAF can utilize machine learning models for data analysis. 3GPP separates the training and inference functions of NWDAF. An NWDAF can support only model training, only data inference, or both. An NWDAF supporting model training can also be called a training NWDAF, or an NWDAF supporting model training logical function (MTLF) (abbreviated as MTLF). A training NWDAF can train a model based on the acquired data to obtain a trained model. An NWDAF that supports data inference can also be called an inference NWDAF, or an NWDAF that supports analytics logical function (AnLF) (abbreviated as AnLF). An inference NWDAF can input input data into a trained model to obtain analysis results or inference data. In this embodiment, a training NWDAF refers to an NWDAF that at least supports model training. As a possible implementation, a training NWDAF can also support data inference. An inference NWDAF refers to an NWDAF that at least supports data inference. As a possible implementation, an inference NWDAF can also support model training. If an NWDAF supports both model training and data inference, it can be called a training NWDAF, an inference NWDAF, a training-inference NWDAF, or simply an NWDAF. In this embodiment, an NWDAF can be a separate network element or co-located with other network elements, such as being placed in a PCF or AMF.

[0149] For example, NWDAF can interact with different network entities to achieve the following different purposes:

[0150] (1) Data collection based on event subscriptions provided by AMF, SMF, UPF, PCF, UDM, the network slice admission control function (NSACF), AF (directly or through NEF), and operation administration and maintenance (OAM);

[0151] (2) Use the data collection coordination function (DCCF) for analysis and data collection;

[0152] (3) Retrieve information from the data repository (e.g., obtain user-related information via UDM, or obtain packet flow description (PFD) information via NEF (packet flow descriptions function, PFDF)).

[0153] (4) Collect location information from the location service (LCS) system;

[0154] (5) Store and retrieve information from the analytics data repository function (ADRF);

[0155] (6) Collect analytics and data from the messaging framework adaptor function (MFAF);

[0156] (7) Retrieve information about NF (e.g., obtain NF-related information from NRF);

[0157] (8) Provide analytics to consumers on demand;

[0158] (9) Provide bulk data related to the Analytics ID(s);

[0159] (10) Provide information on the accuracy of the analysis ID;

[0160] (11) Provide information about the accuracy of the ML model or the decrease in the accuracy of the ML model.

[0161] like Figure 2As shown, the 5G system architecture allows any 5G core network (5GC) NF to request network analysis information from the NWDAF containing the AnLF. The NWDAF and the 5GC NF using the analysis information belong to the same PLMN. The Nnwdaf interface is defined for the 5GC NF and is used to request subscription to network analysis deliveries for a specific context, unsubscribe from network analysis deliveries, and request specific reports of network analysis for a specific context.

[0162] The current 3GPP standard 23288 defines the ability of NWDAF to collect user service access data and signaling data from NF and perform data analysis through service-based interfaces or private interfaces. Table 1 below shows the NF services provided by NWDAF. The same service name can correspond to different service operations and operation semantics, and can also be applied to different NF consumers.

[0163] Table 1

[0164]

[0165]

[0166] 12. Application Provider (AF): This relays the application's requests to the network, such as QoS requirements or user state event subscriptions. AF can be a third-party functional entity or an application server deployed by the operator.

[0167] It should be understood that the above examples illustrate several core network elements included in the CN. In addition, other core network elements may be included. For example, in a machine learning (ML) scenario, the CN may also include ADRF (Advanced Digital Retrieval System). Figure 1 (Not shown in the image), this ADRF can be used to store model-related data. The data can be generated by AnLF, and the ADRF can provide model-related data to MTLF upon request.

[0168] It is understandable that the aforementioned network elements or functions can be physical entities in hardware devices, software instances running on dedicated hardware, or virtualization functions instantiated on a shared platform (e.g., a cloud platform). Simply put, an NF can be implemented in hardware or software.

[0169] A Data Network (DN) is a network located outside of the carrier's network. A carrier's network can connect to multiple DNs, and various services can be deployed on a DN, providing data and / or voice services to terminal devices. For example, a DN might be the private network of a smart factory. Sensors installed in the workshop can act as terminal devices, and a control server for these sensors is deployed within the DN. The control server provides services to the sensors. Sensors can communicate with the control server, receive instructions from it, and transmit the collected sensor data back to the control server accordingly. Another example is a DN serving as the internal office network of a company. Employees' mobile phones or computers can act as terminal devices, accessing information and data resources on the company's internal office network.

[0170] Figure 1 Nnssf, Nnef, Nnrf, Npcf, Nudm, Nudr, Nnwdaf, Naf, Nausf, Namf, Nsmf, N1, N2, N3, N4, and N6 are interface sequence numbers. For example, the meaning of the above interface sequence numbers can be found in the definitions in the 3GPP standard protocols, and this application does not limit the meaning of the above interface sequence numbers. It should be noted that... Figure 1 The interface names between the various network functions in this document are merely examples. In specific implementations, the interface names of this system architecture may be other names, and this application does not limit them. Furthermore, the names of the messages (or signaling) transmitted between the aforementioned network elements are also merely examples and do not constitute any limitation on the function of the messages themselves.

[0171] It should be noted that, in Figure 1 In the architecture shown, the interface between (R)AN and CN can also be called the NG interface. Figure 1 (Not shown in the diagram), (R)AN and CN are connected via the NG interface. The NG interface may include the NG-C interface and the NG-U interface. The NG-C interface is the control plane interface, connecting (R)AN and AMF, and is used to transmit control plane data. The NG-U interface is the user plane interface, connecting (R)AN and UPF, and is used to transmit user plane data.

[0172] It should be understood that Figure 1 The AMF, SMF, UPF, NEF, AUSF, NRF, PCF, and UDM shown can be understood as network elements in the core network used to implement different functions, such as network slices that can be combined as needed. These core network elements can be independent devices or integrated into the same device to implement different functions. This application does not limit the specific form of the above network elements.

[0173] It should also be understood that the above naming is defined solely for the purpose of distinguishing different functions and should not constitute any limitation on this application. This application does not preclude the possibility of using other naming conventions in 5G networks and other future networks. For example, in future communication networks, some or all of the above-mentioned network elements may use the terminology from 5G, or they may use other names, etc.

[0174] To advance the intelligent evolution of the core network, 3GPP has defined NWDAF, which introduces fundamental AI capabilities such as data acquisition, model training, and inference analysis into the network operation mechanism. Based on the 3GPP definition, NWDAF can be deeply integrated with intelligent service operation, realizing real-time data acquisition, intelligent sensing, analysis and inference, and decision-making loops on a user-by-user and service-by-service basis, providing new ideas for intelligent service experience perception and assurance.

[0175] In various intelligent scenarios defined by 3GPP, such as user service perception, service quality assessment, service experience upgrade, personalized service recommendation, and user behavior prediction, the current methods for obtaining labeled data samples to perform model training and model verification tasks are based on a combination of human resources and tools, including steps such as dialing, packet capture, data analysis and labeling, network data collection, and result comparison.

[0176] like Figure 3 As shown, an AI model that has undergone offline training and model release (Phase 1) needs to go through three stages before it can be deployed or redeployed on existing network devices: data preparation (Phase 2), model fine-tuning (Phase 3), and model management and deployment (Phase 4). The key actions in the data preparation stage are purchasing typical market terminal devices, compiling a list of high-frequency applications (APPs), determining the duration and content of test samples (e.g., videos, web pages), classifying and labeling each sample file, and converting messages into structured labeled samples (data preprocessing). However, the method based on a combination of manpower and tools has the following service drawbacks: how to obtain the list of terminal devices and APPs, how to set the frequency and duration of test samples, the high skill requirements for personnel in file sampling and labeling, how to maintain the test system, and how to manage the local test scripts.

[0177] The key actions in the model fine-tuning stage are: performing data mining on NWDAF based on the labeled data from the probing test to obtain more data samples from the live network (sample data mining); judging the validity of the sample data and whether the data size reaches the threshold to perform fine-tuning tasks (fine-tuning task execution); and evaluating whether the overall accuracy of the new template and the application accuracy of this probing test meet the requirements, which may require iterative probing tests to obtain samples (model training effect evaluation). However, the method based on a combination of manpower and tools has the following service drawbacks: how to judge whether the execution effect of the sample data mining task has met expectations, how to evaluate whether the model has met the requirements after fine-tuning training, and how to handle the situation where the model effect does not meet expectations after multiple rounds of probing tests.

[0178] The key actions in the model management and deployment phase are the generation and release of new models (release) and the loading of new models onto existing network devices (loading). However, the approach based on a combination of manpower and tools has the following service drawbacks: after the release of a new model, it involves the existing network, making it difficult to streamline customer-side processes and lacking a defined quality process.

[0179] Therefore, in intelligent scenarios, how to automatically verify the usability of models (such as whether there are deviations in the model inference results), and how to automate the implementation of APP testing, packet capture and data annotation, sample acquisition and training tasks when the usability of AI models deployed on the live network deviates, are urgent research topics.

[0180] To this end, this application provides a communication method and apparatus to establish a channel for authentication, control commands, and annotation data exchange between the end-side automatic probing system and the NWDAF deployed in the network. This enables automated verification of model availability, construction of sample sets, and execution of model training / fine-tuning tasks in intelligent scenarios.

[0181] The technical terms used in the embodiments of this application are described below.

[0182] Dial-up testing: refers to the process by which a terminal device initiates a business access process to an application by launching the application's business access script.

[0183] Data cleaning is a crucial step in data preprocessing. It primarily involves removing "dirty data" by filling in missing values, smoothing noisy data, identifying or removing duplicate data, and correcting errors, thus ensuring data quality and effectiveness. The goal of data cleaning is to ensure that the data is accurate, consistent, and error-free, as incorrect or inconsistent data can negatively impact model performance.

[0184] Data labeling plays a crucial role in model training. It involves marking, classifying, and labeling data to help machine learning algorithms understand and generate accurate predictions. The purpose of data labeling is to transform raw data into a format that the model can understand and use, thereby improving the model's performance and accuracy.

[0185] Please see Figure 4 , Figure 4 This is a schematic diagram of the architecture of a communication system used in an embodiment of this application. As an example, such as... Figure 4 As shown, the communication system includes: a first automatic dialing test system, a user plane network element, a first device, and a data analysis network element. The three can communicate directly or indirectly with each other, without limitation.

[0186] The first automatic dial-up testing system is deployed on the terminal device side, supporting tasks such as dial-up testing, data collection and labeling, and traffic statistics issued by the data analysis network element. It drives the dial-up testing terminal device to trigger corresponding application service access and supports data cleaning and labeling tasks on the terminal side. Then, it reports the relevant labeled data to the data analysis network element through different paths. In other words, the first automatic dial-up testing system is used to implement functions such as application dial-up testing, dial-up data collection, cleaning, labeling, and labeled data reporting. In this embodiment, the first automatic dial-up testing system includes at least one terminal device for dial-up testing applications. In subsequent embodiments, the first terminal device is used as an example. For instance, the first terminal device in the first automatic dial-up testing system initiates the service access process for the first application by launching an application dial-up testing script for the first application, thereby enabling dial-up data collection, dial-up data packet capture, dial-up data cleaning and traffic statistics, dial-up data collection and labeling, and labeled data reporting.

[0187] Optionally, the first automatic dialing test system may further include an automatic dialing test control device for controlling terminal devices of dialing test applications. This automatic dialing test control device can control one or more terminal devices of dialing test applications. The automatic dialing test control device can be used to control the terminal devices to perform operations such as dialing tests and dialing test data acquisition, and can also perform data cleaning, traffic statistics, and data labeling based on the application data collected from the terminal devices. For example, the automatic dialing test control device can be a personal computer, a PC, etc.

[0188] In some scenarios, the terminal equipment in the automatic dialing test system includes an automatic dialing test control device, or the terminal equipment has a module with the function of an automatic dialing test control device. In this case, the first automatic dialing test system may not include an automatic dialing test control device, but only include terminal equipment used for dialing test applications.

[0189] The user plane network element is deployed with a first model, which can be used for business-related inference, such as business identification and business quality assessment. The user plane network element can collect business flow data and report business flow-related information (such as feature data and quintuple information) during the business access process of the application tested by the first automatic dialing test system. For example, the user plane network element can be the aforementioned... Figure 1 The UPF in the 5G architecture is shown.

[0190] The first device is used to establish a secure channel between the first automatic dial-up testing system and the data analysis network element to support the implementation of functions such as task distribution and labeled data reporting. In this embodiment, the first device can be a user plane network element, a dial-up testing server, or a security bastion host. The dial-up testing server refers to an independently configured server located on the network side for serving the automatic dial-up testing system; it can be called an automatic dial-up testing server, a dial-up testing data acquisition server, etc., without limitation.

[0191] A security bastion host, also known as a security operations and maintenance management platform, is a network security device used for security authentication and access authorization of automatic dial-up testing systems. It improves secure communication between the automatic dial-up testing system and data analysis network elements. The security bastion host is wired to the automatic dial-up control device in the automatic dial-up testing system.

[0192] The data analysis network element supports end-to-end control of the automated testing system, sample feature acquisition, and AI inference result comparison and analysis, enabling fully automated AI inference accuracy verification and sample acquisition capabilities. For example, the data analysis network element is used for functions such as issuing the first task related to the first model, generating sample data, model training, and deployment. Furthermore, the data analysis network element supports building secure authentication mechanisms and communication channels with the end-side automated testing system through multiple paths, including user plane network elements, testing servers, and security bastion hosts. For instance, the data analysis network element can be used for the aforementioned... Figure 1 The NWDAF in the 5G architecture shown here will not be elaborated on in detail here.

[0193] In this embodiment, after the data analysis network element establishes a secure channel with the first automatic testing system via the first device, the data analysis network element can send a first task request to the user plane network element and a second task request to the first device. The first task request requests the user plane network element to acquire and report first data of the service flow accessing the first application by the first automatic testing system. The second task request requests the first device to trigger the first automatic testing system to acquire and report labeled data of the service flow accessing the first application. This allows the acquisition of the first data and labeled data to complete the first task related to the service inference or training of the first model. Thus, the availability verification of the model deployed in the existing network, sample set construction, and model training / fine-tuning tasks can be automatically implemented.

[0194] The following will combine Figures 5-8 The communication method provided in the embodiments of this application will be described in detail.

[0195] For example, Figure 5 This is a flowchart illustrating a communication method provided in an embodiment of this application. The communication method uses... Figure 4The communication between the first automatic dialing test system, the user plane network element, the first device, and the data analysis network element is illustrated using the example shown. Of course, the entity executing the action of the first automatic dialing test system in this method can also be a device / module within the first automatic dialing test system, such as a chip, processor, or processing unit within the first automatic dialing test system; there is no limitation on this. Similarly, the entity executing the action of the user plane network element in this method can also be a device / module within the user plane network element, such as a chip, processor, or processing unit within the user plane network element; there is no limitation on this. Likewise, the entity executing the action of the first device in this method can also be a device / module within the first device, such as a chip, processor, or processing unit within the first device; there is no limitation on this. Finally, the entity executing the action of the data analysis network element in this method can also be a device / module within the data analysis network element, such as a chip, processor, or processing unit within the data analysis network element; there is no limitation on this.

[0196] The communication method provided in this application is executed when the data analysis network element establishes a secure channel with the first automatic dialing system through the first device, or in other words, after the secure channel is established. The specific implementation of how the data analysis network element establishes a secure channel with the first automatic dialing system through the first device can be found in the relevant descriptions in the following embodiments, and will not be repeated here.

[0197] like Figure 5 As shown, the communication method includes:

[0198] S501, the data analysis network element sends a first task request to the user plane network element. Correspondingly, the user plane network element receives the first task request from the data analysis network element.

[0199] The first task request is used to request the user plane network element to obtain and report the first data of the service flow of the first application accessed by the first automatic dialing test system. The first data is used to complete the first task related to the service inference or training of the first model. The first model is used for service-related inference.

[0200] In other words, the first task request is related to the first task, the first model is deployed in the network, such as the first model trained by the data analysis network element is deployed on the user plane network element, which is the first model used for inference, and the first task is the task related to the inference or training of the first model used for business-related inference.

[0201] In this embodiment, the first task may be to verify whether the business reasoning result of the first model meets expectations, or the first task may be to generate a sample set for training the first model. Verifying whether the business reasoning result of the first model meets expectations can refer to verifying the accuracy or reliability of the business reasoning result of the first model, which is a verification of the usability of the first model. Generating a sample set for training the first model can refer to obtaining a sample set for training the first model because the business reasoning result does not meet expectations, in order to complete model fine-tuning so that the model reasoning result meets expectations. This can also be called a sample collection task, a model training task, or a model fine-tuning task, without limitation. It can be understood that the first task is either a usability verification of the first model or a fine-tuning of the first model.

[0202] The following examples illustrate intelligent SA and intelligent QoE scenarios for business experience evaluation:

[0203] In the intelligent SA scenario, the business inference result of the first model is the business identification result. This business identification result can be the application type or application type + business type that the user accesses, obtained through inference by the first model during the user's access to the business. Therefore, the first task can be to verify whether the recognition rate (referred to as the business recognition rate) of the business identification result obtained by the first model reaches the recognition rate threshold. If the recognition rate threshold is not reached, the data analysis network element can train the first model to adjust the business recognition rate of the first model to meet the requirements.

[0204] In the intelligent QoE scenario, the business inference result of the first model is the business quality assessment result. This business quality assessment result can be used to evaluate the user's experience quality when accessing services through the application, and can also be called the application or service's (quality of experience, QoE) assessment result. For example, in a video application, the business quality assessment result may include vMOS results and / or KQI assessment results. The business quality assessment result output by the first model inference can be measured by a score. Therefore, the first task can be to verify whether the assessment accuracy of the business quality assessment result obtained by the first model inference reaches the accuracy threshold. If it does not reach the accuracy threshold, the data analysis network element can also train the first model to adjust the business quality assessment result of the first model to meet the requirements.

[0205] It is understandable that if the usability verification results of the first model do not meet expectations, the data analysis network element needs to obtain samples for model training to train the first model. Therefore, the first task can also be to generate a sample set for training the first model. In the two scenarios in the above example, the sample set can be used to train the first model's business recognition capability or business quality assessment capability. This sample set can include a training set and a validation set.

[0206] After the data analysis network element establishes a secure channel with the first automatic dialing system through the first device, the data analysis network element can periodically start the first task, start the first task at regular intervals, or start the first task immediately. Based on the first task, it can send different task requests to the user plane network element and the first device to perform verification or training tasks on the first model deployed in the network.

[0207] The first task request corresponds to the task to be executed by the user plane network element, namely, to obtain and report the first data of the service flow of the first automatic dialing test system accessing the first application. It can be considered as the data analysis network element obtaining the first task, or the task to be executed by the user plane network element is one or more sub-tasks obtained by splitting the first task. In the embodiments of this application, the first task may include sub-tasks executed for the user plane network element (i.e., the sub-tasks to be executed corresponding to the first task request) and sub-tasks executed for the first automatic dialing test system (i.e., the sub-tasks to be executed corresponding to the second task request below).

[0208] In this embodiment of the application, the first task request may include one or more of the following: the identifier of the first automatic dialing system (such as the identifier of the first terminal device), the identifier of the first application (application identifier or application script identifier), the type or identifier of the first task, the type or identifier of the subtask related to the first task, task requirement information, etc., without limitation.

[0209] After receiving the first task request, the user plane network element initiates the collection and reporting of the service flow of the first application accessed by the first automatic dialing test system. During the process of the first automatic dialing test system dialing the first application and accessing the service flow of the first application through the user plane, the user plane network element can collect the service flow of the first application and obtain the statistical characteristic information of the service flow based on the collected service flow.

[0210] The first automatic dialing test system includes a first terminal device for dialing a first application. The first application is an application installed or downloaded on the first terminal device used for dialing in the first automatic dialing test system. The data analysis network element triggers the first terminal device in the first automatic dialing test system to dial the first application through a first device, which enables the user plane network element to obtain first data and the first automatic dialing test system to obtain labeled data, thereby enabling the data analysis network element to complete the first task by obtaining the first data and labeled data. In this embodiment, the first automatic dialing test system accessing the first application generally refers to the first terminal device in the first automatic dialing test system accessing the first application.

[0211] The first data refers to the service flow-related data acquired by the user plane network element during application testing in the first automatic dialing test system, based on the collected service flow, to achieve the first task. In this embodiment, the first data may also be referred to as service data, access data, user access data, service access data, service flow experience data, etc., without limitation. It should be understood that the data type of the first data may differ depending on the first task. A detailed description of the labeled data can be found in S502 below.

[0212] If the first task is to verify whether the service inference result of the first model meets expectations, then the first data is used to compare with the labeled data to verify whether the service inference result obtained by the user plane network element using the first model is the same as the actual service result used as a label in the labeled data obtained by the end-side automatic dialing test system, thereby verifying whether the service inference result of the first model meets expectations. The first data may include the service inference result, service flow access data obtained by the user plane network element from the service flow of the collected first application, and five-tuple information. The service inference result is obtained by the user plane network element using the first model to infer the service flow access data obtained by the user plane network element. The five-tuple information includes the source IP address, source port, destination IP address, destination port, and transport layer protocol of the service flow.

[0213] For example, the first application is a video application. During the process of the user plane network element accessing the service flow of the first application in the first automatic dialing test system, the service flow access data is collected. The service flow access data may include the traffic of the service flow, the message content of part of the service flow, etc. The user plane network element will input the service flow access data into the first model, which can infer which video application or which type of service in the video application is being accessed (such as live streaming, watching live streaming, short video, news or long video). The service inference result can be the application identifier or the application identifier + service identifier.

[0214] If the first task is to generate a sample set for training the first model, then the first data is used to associate with the labeled data. The first data is the business flow feature data used to generate the sample data from the labeled data; in other words, the labels in the labeled data serve as the labels in the sample data, and the business flow feature data in the first data serves as the training data in the sample data. Specifically, the first data may include business flow feature data and quintuple information collected from the business flow of the first application by the user plane network elements. In this case, the first data is extracted by the user plane network elements from the collected business flow of the first application and is not obtained through the first model.

[0215] For example, the first application is a video application. The service flow characteristic data may include service flow statistical characteristics and service flow load characteristics. Service flow statistical characteristics may include the packet length sequence, protocol type, transmission rate, uplink and downlink traffic ratio, mean, variance, etc. of the first N (N is a positive integer) service flow packets. Service flow load characteristics may include the load size of the service flow, etc.

[0216] Optionally, after the user plane network element accepts or initiates the task of acquiring and reporting the first data according to the first task request, it can send a first task response to the data analysis network element to inform or notify the data analysis network element that it has accepted or initiated the task of acquiring and reporting the first data. The first task response can be a response message corresponding to the first task request.

[0217] S502, the data analysis network element sends a second task request to the first device. Correspondingly, the first device receives the second task request from the data analysis network element.

[0218] The second task request is used to request the first device to trigger the first automatic dialing system to obtain and report the labeled data of the business flow accessing the first application. The labeled data is also used to complete the first task related to the business reasoning or training of the first model.

[0219] After the data analysis network element initiates the first task, in addition to sending the first task request to the user plane network element, it also sends a second task request to the first device. This second task request triggers the first automatic testing system to test the first application and obtain labeled data. Similar to the first task request, the second task request is the task that the first device requests to execute, namely, triggering the first automatic testing system to obtain and report labeled data of the service flow accessing the first application. This data is obtained by the data analysis network element based on the first task, or one or more sub-tasks obtained by breaking down the first task.

[0220] In this embodiment, the first device triggering the first automatic dial-testing system to acquire and report labeled data of the service flow accessing the first application may include: the first device triggering the first automatic dial-testing system to start dial-testing of the first application, dial-testing data packet capture, dial-testing data cleaning and traffic statistics, dial-testing data collection and labeling, labeled data reporting, and the first device starting to monitor and report the labeled data of the first automatic dial-testing system. Therefore, the data analysis network element requests the first device to perform the above tasks to obtain the labeled data of the first automatic dial-testing system.

[0221] Among them, the tasks of the first automatic dialing test system initiating dialing test on the first application, dialing test data packet capture, dialing test data cleaning and traffic statistics, dialing test data collection and labeling, and labeling data reporting can all be collectively referred to as the dialing test tasks of the first automatic dialing test system, without any limitation.

[0222] In this embodiment of the application, the second task request may include one or more of the following: the identifier of the first device, the identifier of the first automatic dialing system, the identifier of the first application, the identifier or type of the first task, the type or identifier of the subtask related to the first task, and task requirement information (such as the execution time of the dialing task), etc., without limitation.

[0223] Annotated data refers to user access data that has been annotated or tagged to achieve the first task, obtained by the first automatic probing system during application probing when the user accesses the first application through the user plane. It should be understood that the type of tag used for annotation may vary depending on the first task.

[0224] Taking the above two scenarios as examples, if the business inference result is a business identification result or the sample set is used to train the business identification capability of the first model, the labeled data may include the application identifier as a label and the business flow access data and quintuple information associated with the application identifier; if the business inference result is a business quality assessment result or the sample set is used to train the business quality assessment capability of the first model, the labeled data may include the business quality result as a label and the business flow access data and quintuple information associated with the business quality result.

[0225] For example, if the first application is a video application, and the business inference result is the business identification result, then the application identifier in the labeled data is the application identifier of the first application. The business flow access data may include the uplink and downlink traffic statistics of the business flow. If the sample set is used to train the business quality assessment capability of the first model, then the business quality result in the labeled data may be a QoE score or a KQI score. The business flow access data may include QoE or KQI information such as resolution, bitrate, first frame loading time, first frame loading time scaling rate, stuttering time, stuttering times, video playback success rate, start-up download rate, media data transmission round-trip latency, media data transmission packet loss rate, and download rate, as well as traffic statistics and packet count.

[0226] In this embodiment of the application, the first device can have the following three designs:

[0227] Design 1: The first device is a user plane network element. In this case, the first task request and the second task request can be considered as the same task request. However, the task request is not only used to request the first data, but also to request the annotation data. In other words, the first task request and the second task request are sent in the same message. No limitation is made in this regard.

[0228] Design 2: The first device is a dial-up test server independently set up in the network. For a detailed description of the dial-up test server, please refer to the relevant description in the above communication system. It will not be repeated here.

[0229] Design 3: The first device is a security bastion host. For a detailed description of the security bastion host, please refer to the relevant description in the above communication system. It will not be repeated here.

[0230] For Design 2 and Design 3, the first task request and the second task request are different task requests.

[0231] Optionally, after receiving the second task request, the first device may also send a corresponding response message to the data analysis network element to instruct it to accept the task.

[0232] S503, the first device sends a third task request to the first automatic testing system according to the second task request. Correspondingly, the first automatic testing system receives the third task request from the first device.

[0233] The third task request is used to request the first automatic testing system to obtain and report the labeled data of the business flow accessing the first application.

[0234] After receiving the second task request, the first device can determine a third task request based on the second task request to trigger the first automatic dialing test system to dial the first application, capture dialing test data packets, clean dialing test data and perform traffic statistics, collect and label dialing test data, and report labeled data. Furthermore, the first device also initiates monitoring and reporting of the labeled data of the first automatic dialing test system based on the second task request.

[0235] The third task request may include one or more of the following: the identifier of the first automatic testing system, the identifier of the first application, the identifier or type of the first task, the type or identifier of the subtask related to the first task, and task requirement information (such as the execution duration of the testing task), etc., without limitation.

[0236] Since the first automatic dialing test system includes a first terminal device for dialing test applications, in some implementations, S503 can be replaced by: the first device sending a third task request to the first terminal device in the first automatic dialing test system according to the second task request, and correspondingly, the first terminal device receiving the third task request from the first device.

[0237] Optionally, after receiving the assigned task, the first automatic dialing system may also send a corresponding response message to the first device to instruct it to accept or start the task.

[0238] S504, the first automatic testing system sends annotation data to the first device. Correspondingly, the first device receives the annotation data from the first automatic testing system.

[0239] After the first automatic dialing test system receives the third task request, the first terminal device in the first automatic dialing test system starts the business access script for the first application, triggering the business flow access process for the first application. That is, it accesses the business flow with the application server of the first application through the user plane, and captures the business flow (or message) accessed by the user during the business flow access process. After the access is completed, the captured business flow is processed to obtain the annotation data, and the obtained annotation data is sent to the first device.

[0240] In the design of the first automatic dialing test system, which also includes an automatic dialing test control device for controlling the dialing test of the first terminal device, after receiving a third task request, the first terminal device sends the third task request to the automatic dialing test control device. That is, the first terminal device sends a third task request to the automatic dialing test control device, and correspondingly, the automatic dialing test control device receives the third task request from the first terminal device. Then, the automatic dialing test control device sends dialing instructions and acquisition instructions to the first terminal device according to the third task request, triggering the first terminal device to start a business access script for the first application. During the business flow access process, it captures the user's accessed business flow (or messages) and feeds back the captured business flow to the automatic dialing test control device. Furthermore, the automatic dialing test control device can process the captured business flow (such as data cleaning, statistics, and annotation) to obtain annotated data, and send an annotation data reporting instruction to the first terminal device, so that the first terminal device sends the annotation data to the first device.

[0241] In this design, the automatic dialing control device triggers the dialing application of the first terminal device and collects dialing data, and generates annotation data based on the dialing data collected by the first terminal device.

[0242] It should be understood that if the first terminal device itself has the function of automatic dialing control, then the application dialing, the collection of dialing data, and the processing of dialing data are all triggered locally by the first terminal device.

[0243] For a detailed description of the labeled data, please refer to the relevant description in S502 above, which will not be repeated here.

[0244] Alternatively, S504 can be: the first terminal device sends annotation data to the first device. Correspondingly, the first device receives the annotation data from the first terminal device.

[0245] S505, the user plane network element sends the first data to the data analysis network element. Correspondingly, the data analysis network element receives the first data from the user plane network element.

[0246] During the service flow of the first application accessed by the user plane in the first automatic dialing test system (or the first terminal device), the user plane network element collects the service flow accessed by the user and processes the collected service flow (such as statistics, feature extraction, and / or inference) to obtain first data. The first data can be used to compare with labeled data to determine whether the data obtained by the user plane network element matches the data actually obtained by the first automatic dialing test system, or the first data can be used to correlate with labeled data to generate sample data with the data actually obtained by the first automatic dialing test system. A detailed description of the first data can be found in the relevant description in S501 above, and will not be repeated here.

[0247] S506, the first device sends labeled data to the data analysis network element. Correspondingly, the data analysis network element receives the labeled data from the first device.

[0248] After the first device obtains the annotation data reported by the first automatic dialing system (or the first terminal device), it sends the annotation data to the data analysis network element.

[0249] When the first device is a user plane network element, optionally, the first data and the annotation data can be sent together, such as in the same message, or they can be sent separately, such as in different messages; there is no limitation on this.

[0250] Correspondingly, after the data analysis network element obtains the first data and the labeled data, it can compare or correlate the two to achieve the first task, such as the usability verification of the first model or the generation of a sample set.

[0251] For example, the first application is a video application, and the first task is to verify whether the service recognition rate obtained by the first model inference reaches the recognition rate threshold. The first data may include the traffic or total traffic of the service flow, five-tuple information, and the application type obtained by inference based on the first model. The labeled data may include the application identifier of the first application, as well as the five-tuple information associated with the application identifier of the first application and the uplink and downlink traffic statistics of the service flow. Thus, the data analysis network element can compare the first data reported by the user plane network element with the labeled service access data (i.e., labeled data) reported by the first automatic dialing test system to confirm whether the service recognition rate of the first model meets the expectations. If it does not meet the expectations, the first task related to sample collection and model fine-tuning can be initiated.

[0252] For example, the first application is a video application. The sample set can be used to train the service recognition capability of the first model. The first data may include the application identifier of the service flow, five-tuple information, the packet length sequence of the first N packets, protocol type, uplink and downlink traffic ratio, mean, variance and other service flow statistical characteristics, and service flow load information. The labeled data may include the application identifier of the first application, as well as the five-tuple information associated with the application identifier of the first application and the uplink and downlink traffic statistics of the service flow. Thus, the data analysis network element can associate the first data reported by the user plane network element and the labeled service access data (i.e., labeled data) reported by the first automatic dialing system to generate sample data containing sample features. Based on the sample data, a sample set is constructed to train the first model and improve the service recognition rate.

[0253] It should be understood that in the embodiments of this application, sample data includes training data and labels. Training data serves as input data for model training, and labels are used to compare with the output data of the model training, and are used to converge the model. For example, training data and labels can form one or more sets of the form (x... i y i The sample data is denoted as , where i is the sample data index and x is the sample data index. i For training data, y i For labeled data, during model training, this y i It can be used with model input x i The output data is compared with the data, and the trainable parameters in the model are adjusted based on the comparison results using methods such as backpropagation and gradient descent to obtain a model for inference.

[0254] Therefore, after establishing a secure channel with the first automatic testing system via the first device, the data analysis network element can send task requests related to the first task to both the user plane network element and the first device. The first task is related to the inference or training of the first model used for business-related inference. This triggers the user plane network element to acquire and report first data related to the service flow access of the end-side application, and triggers the first automatic testing system via the first device to perform application testing, acquiring and reporting labeled data related to the service flow access of the application. Thus, the first task is achieved based on the first data and labeled data. This enables automated model verification, sample collection, and model fine-tuning in intelligent scenarios.

[0255] It should be understood that in the embodiments of this application, the first task is a task that relies on the first automatic testing system on the edge to test the application, and is used for model evaluation and verification, model training and fine-tuning.

[0256] The above describes how the data analysis network element uses the secure channel established by the first device to achieve task distribution and data interaction with the first automatic dialing system. The following explains how the data analysis network element uses the first device to establish a secure channel with the first automatic dialing system.

[0257] The data analysis network element establishes a secure channel with the first automatic dialing system through the first device, which may include:

[0258] When the data analysis network element completes the security authentication of the first automatic dialing test system through the first device, the data analysis network element establishes a secure channel with the first automatic dialing test system through the first device. The secure channel supports the data analysis network element in issuing tasks to the first automatic dialing test system and supports the first automatic dialing test system in reporting labeled data.

[0259] In other words, the data analysis network element performs security authentication on the first automatic dialing test system through the first device. If the security authentication is successful, the data analysis network element can establish a secure channel with the first automatic dialing test system through the first device and execute the above-mentioned S501 to S506.

[0260] The data analysis network element completes the security authentication of the first automatic dialing test system through the first device. The following is a description of the three designs of the first device mentioned above:

[0261] Design 1: The first device is a user plane network element. In this case, the user plane network element can act as a dial-up test server to provide services to the automatic dial-up test system on the end side.

[0262] In this design 1, the data analysis network element sends a first authentication request to the user plane network element, and correspondingly, the user plane network element receives the first authentication request from the data analysis network element. The first authentication request is used to request the user plane network element to perform security authentication on the automatic dialing test system to be monitored. The first authentication request includes the first authentication information of the automatic dialing test system to be monitored, and the automatic dialing test system to be monitored includes the first automatic dialing test system.

[0263] In other words, the data analysis network element configures the authentication information of the automatic dialing test system to be monitored, namely the first authentication information, and sends the first authentication information to the user plane network element in the first authentication request, so as to request the user plane network element to perform security authentication on the automatic dialing test system to be monitored based on the first authentication information.

[0264] The first authentication information may include the authentication information of the terminal device used for testing in the automatic dialing test system to be monitored. The authentication information of the terminal device may include the terminal device's identifier, the terminal device's authentication account, and the terminal device's authentication key. For example, the terminal device's identifier may be the International Mobile Subscriber Identity (IMSI) or the Mobile Station International ISDN Number (MSISDN).

[0265] Optionally, the first authentication information may also include the address of the dial-up testing server. The dial-up testing server corresponds to the first automatic dial-up testing system; that is, the dial-up testing server is a server used to serve the first automatic dial-up testing system. In Design 1, this can be understood as the address of the dial-up testing server being a user plane network element. The address of the dial-up testing server can be pre-configured or pre-defined on the user plane network element. In this case, the first authentication information may not include the address of the dial-up testing server.

[0266] In some designs, the address of the test server can also be included in the first authentication information, that is, the test server is used as an authentication parameter and there are no restrictions on it.

[0267] It should be understood that there may be one or more automatic dialing test systems to be monitored. Therefore, each automatic dialing test system to be monitored may correspond to a first authentication information, or the first authentication information may include the authentication information corresponding to one or more automatic dialing test systems to be monitored respectively. The authentication information corresponding to one or more automatic dialing test systems to be monitored may be indicated in the form of a list.

[0268] Optionally, the first authentication request may also include authentication-related information such as the validity period of the authentication information, the number of terminal devices that can be authenticated simultaneously, and the timeout period for each authentication, without limitation.

[0269] The user plane network element initiates monitoring of the automatic dial-up testing system to be monitored based on the first authentication request, saves the first authentication information, and waits for the automatic dial-up testing system on the end side to initiate authentication.

[0270] When the first automatic dial-up testing system initiates authentication, it sends a fourth authentication request to the user plane network element. Correspondingly, the user plane network element receives the fourth authentication request from the first automatic dial-up testing system. This fourth authentication request requests security authentication from the first automatic dial-up testing system and includes the authentication information of the first automatic dial-up testing system. Then, based on the first authentication information and the authentication information of the first automatic dial-up testing system, the user plane network element sends a fourth authentication response to the first automatic dial-up testing system. This fourth authentication response notifies the first automatic dial-up testing system that the security authentication was successful; the fourth authentication response can be a response message to the fourth authentication request.

[0271] In other words, after the user plane network element receives the fourth authentication request from the first automatic dial-up testing system, it compares the authentication information of the first automatic dial-up testing system in the fourth authentication request with the first authentication information. It then determines whether the first automatic dial-up testing system needs to be monitored based on the identifier of the first terminal device in the first automatic dial-up testing system. If so, it checks whether the address of the dial-up testing server in the fourth authentication request is the same as the address of the dial-up testing server stored locally. If they are the same, it checks whether the authentication account and authentication key are the same. If they are the same, the first automatic dial-up testing system is successfully authenticated, and an authentication success response is sent to the first automatic dial-up testing system.

[0272] If the authentication information in the fourth authentication request is inconsistent with the first authentication information of the automatic dialing test system to be monitored stored in the user plane network element, the fourth authentication response can be used to notify the first automatic dialing test system of security authentication failure. Optionally, the fourth authentication response can carry the reason for the security authentication failure.

[0273] Optionally, after the user plane network element successfully authenticates the first automatic dialing test system, the user plane network element can also send a first authentication response to the data analysis network element. Correspondingly, the data analysis network element receives the first authentication response from the user plane network element. The first authentication response is used to notify the data analysis network element that the first automatic dialing test system has successfully authenticated the system. The first authentication response includes the identifier of the first terminal device. The first authentication response can be a response message to the first authentication request.

[0274] Therefore, after the data analysis network element learns that the first automatic dialing test system has successfully authenticated, it can establish a secure channel through the user plane network element, start the first task, and execute the above S501 to S506.

[0275] Design 2: The first device is a dial-up test server, which is set up independently.

[0276] In this design 2, the data analysis network element can first perform security authentication on the dial-up testing server. If the security authentication is successful, the dial-up testing server will then perform security authentication on the first automatic dial-up testing system that initiated the authentication.

[0277] The data analysis network element can perform security authentication on the testing server. This can include: the testing server sending a second authentication request to the data analysis network element; the data analysis network element receiving the second authentication request from the testing server; and the second authentication request including the testing server's authentication information. Subsequently, the data analysis network element sends a second authentication response to the testing server; and the testing server receiving the second authentication response from the data analysis network element. The second authentication response is used to notify the testing server that the security authentication was successful.

[0278] The data analysis network element is configured with authentication information for the test server to be authenticated. This authentication information may include the test server's address, authentication account, and authentication key. When the data analysis network element receives a second authentication request from the test server, it can perform security authentication on the test server initiating the authentication based on the authentication information in the second authentication request. If the authentication information of the test server in the second authentication request matches the authentication information of the test server to be authenticated configured locally by the data analysis network element, the test server is successfully authenticated, and the data analysis network element sends an authentication success response to the test server. Otherwise, if they do not match, the data analysis network element sends an authentication failure response to the test server.

[0279] It is understandable that the second authentication response can be a response message to the second authentication request, which can be used to notify the test server whether the security authentication was successful or failed.

[0280] If the test server is successfully authenticated, it waits for the automatic test system on the end side to initiate authentication. At this time, the test server can configure the authentication information of the test server to be monitored locally, or it can be issued by the data analysis network element as described in Design 1 above. There are no restrictions on this.

[0281] When the first automatic dial-up testing system initiates authentication, it sends a fourth authentication request to the dial-up testing server. Correspondingly, the dial-up testing server receives the fourth authentication request from the first automatic dial-up testing system. This fourth authentication request requests security authentication of the first automatic dial-up testing system and includes the authentication information of the first automatic dial-up testing system. The process by which the dial-up testing server performs security authentication on the first automatic dial-up testing system is similar to the process by which the user plane network element performs security authentication on the first automatic dial-up testing system in Design 1 above, and will not be elaborated upon further.

[0282] Optionally, if the dial-up testing server successfully authenticates the first automatic dial-up testing system, the dial-up testing server may also send an authentication notification to the data analysis network element. This authentication notification is used to notify the first automatic dial-up testing system that the security authentication has been successful.

[0283] Therefore, after the data analysis network element learns that the first automatic dial-test system has successfully completed security authentication, it can establish a secure channel through the dial-test server, start the first task, and execute the above S501 to S506.

[0284] Design 3: The first device is a security bastion host. At this time, the data analysis network element directly acts as a dial-up test server. The address of the dial-up test service and the authentication information of the automatic dial-up test system to be monitored are configured to perform security authentication on the end side of the automatic dial-up test system through the security bastion host.

[0285] In this design 3, the first automatic dialing test system also includes an automatic dialing test control device for controlling the dialing test of the first terminal device. When the first automatic dialing test system initiates authentication, the first automatic dialing test system sends a fourth authentication request to the security bastion host. Correspondingly, the security bastion host receives the fourth authentication request from the first automatic dialing test system. The fourth authentication request is used to request the data analysis network element to perform security authentication on the first automatic dialing test system. The fourth authentication request includes the authentication information of the first automatic dialing test system.

[0286] Optionally, the authentication information of the first automatic dialing test system may include authentication information for security bastion host authentication and authentication information for data analysis network element authentication. For example, the authentication information for security bastion host authentication may include the address of the automatic dialing test control device, authentication account 1, and authentication key 1, while the authentication information for data analysis network element authentication may include the address of the dialing test server, the identifier of the first terminal device, authentication account 2, and authentication key 2.

[0287] Subsequently, the security bastion host sends a third authentication request to the data analysis network element based on the authentication information of the first automatic dialing test system. Correspondingly, the data analysis network element receives the third authentication request from the security bastion host. The third authentication request is used to request the data analysis network element to perform security authentication on the first automatic dialing test system, and the third authentication request includes the authentication information of the first automatic dialing test system.

[0288] In other words, the security bastion host authenticates the first automatic dialing test system based on the authentication information of the first automatic dialing test system in the fourth authentication request, confirms that the address symbol requirements, authentication account and authentication key of the automatic dialing test control device are correct, and sends a third authentication request to the data analysis network element if the security authentication of the first automatic dialing test system is successful.

[0289] Correspondingly, after receiving the third authentication request, the data analysis network element performs security authentication on the first automatic dialing test system based on the authentication information carried in the third authentication request. The authentication process is similar to the process of the user plane network element in Design 1 and the dialing test server in Design 2 performing security authentication on the first automatic dialing test system, and will not be described in detail here.

[0290] If the data analysis network element successfully authenticates the first automatic dialing test system, it sends a third authentication response to the security bastion host. The security bastion host receives this third authentication response from the data analysis network element, which notifies the first automatic dialing test system of successful authentication. Further, the security bastion host sends a fourth authentication response to the first automatic dialing test system, which receives this fourth authentication response, also notifying the first automatic dialing test system of successful authentication. The third authentication response can be a response to a third authentication request, and the fourth authentication response can be a response to a fourth authentication request.

[0291] In some scenarios, the third and fourth authentication requests can be the same authentication request. This can be understood as the security bastion host performing security authentication on the first automatic testing system and then forwarding the authentication request to the data analysis network element. Correspondingly, the third and fourth authentication responses can be the same authentication response, which the security bastion host then forwards.

[0292] In this design 3, the first automatic dialing test system sends a fourth authentication request to the security bastion host, which may include: the automatic dialing test control device sending the fourth authentication request to the security bastion host. The first automatic dialing test system receives a fourth authentication response from the security bastion host, which may include: the automatic dialing test control device receiving the fourth authentication response from the security bastion host.

[0293] Therefore, after the data analysis network element successfully authenticates the first automatic dialing test system, it can establish a secure channel through the security bastion host, start the first task, and execute the above S501 to S506.

[0294] As can be seen from the above three designs, the first device receives the authentication request (such as the fourth authentication request mentioned above) of the first automatic dialing test system and performs security authentication on the first automatic dialing test system based on the fourth authentication request.

[0295] Through the above authentication process, after the first automatic dialing test system completes security authentication through the first device, the data analysis network element can build a secure channel with the automatic dialing test system on the end side through the first device. Based on the secure channel built between the automatic dialing test system and the data analysis network element, the automatic dialing test system is driven to access application services, collect data, and annotate data. The data analysis network element completes tasks such as automated verification of intelligent scenarios, sample set construction, and model training / fine-tuning.

[0296] The following section combines 5G networks and specific application scenarios to discuss... Figure 5 The communication method shown is described in detail.

[0297] Taking a video app as the first application, the first task as verifying the service recognition rate obtained from the first model inference, a UPF as the first device, an NWDAF as the data analysis network element, and an automatic dialing test system including a UE and an automatic dialing test control PC as an example. Figure 6 A flowchart illustrating a communication method provided in an embodiment of this application is shown. Figure 6 As shown, the communication method includes:

[0298] S601, NWDAF configures the first authentication information of the automatic dialing test system to be monitored.

[0299] The first authentication information may include the identifier of the automated dial-up testing system to be monitored, such as the UE's IMSI or MSISDN, authentication account, and authentication key. Optionally, the first authentication information may also include the address of the dial-up testing server and other authentication-related information.

[0300] S602, NWDAF sends an automatic dial-test authentication request to UPF. Correspondingly, UPF receives the automatic dial-test authentication request from NWDAF.

[0301] The automatic testing authentication request (Nupf_AutoLabel_Certification Request) is used to request the automatic testing system on the UPF authentication side, or in other words, the automatic testing authentication request is used to send the first authentication information of the automatic testing system to be monitored. The automatic testing authentication request includes the first authentication information of the automatic testing system to be monitored.

[0302] For example, information elements in an automated dial-up authentication request may include:

[0303] --NF ID (identifier of the authentication network element, such as UPF ID)

[0304] --Target of Certification (The user filtering criteria that need to be monitored and certified, such as all users or users who meet specific conditions)

[0305] --(set of)Account Info(s) (List of verified accounts and keys)

[0306] --(Optional) Certification Information (Other key authentication information, such as the validity period of the authentication information, the number of UEs that can be authenticated simultaneously, etc.)

[0307] --(Optional) IP address of the AutoLabel Server (the address of the autoLabel server that needs to perform monitoring, authentication, and interactive tasks)

[0308] --(Optional) Timeout (duration of timeout for each authentication)

[0309] The automatic dialing authentication request corresponds to the first authentication request in Design 1 above. The specific description of the first authentication information can be found in the above embodiment, and will not be repeated here.

[0310] S603, UPF initiates the monitoring task of the automatic dial-up testing system to be monitored.

[0311] After receiving an automatic dial-test authentication request, UPF initiates monitoring and authentication of the specified automatic dial-test system that meets the user's filtering conditions, and records the first authentication information.

[0312] S604. The Automatic Dialing Control PC sends an authentication command to the UE. Correspondingly, the UE receives the authentication command from the Automatic Dialing Control PC.

[0313] After the first automatic dialing test system is started, the automatic dialing test control PC in the first automatic dialing test system can be connected via a universal serial bus (USB) to send authentication commands, instructing the APP (such as a video APP) on the UE to initiate the authentication process through normal user plane data access.

[0314] S605, the UE sends authentication request #1 to the UPF. Correspondingly, the UPF receives authentication request #1 from the UE.

[0315] Among them, authentication request #1 is used to request security authentication of the first automatic dialing test system, and authentication request #1 includes the authentication information of the first automatic dialing test system.

[0316] After receiving the authentication command, the UE initiates the authentication process to the UPF through normal user plane data access, that is, sends authentication request #1. Authentication request #1 carries information such as the dial-up test server address, UE ID, authentication account and authentication key for authentication by the first automatic dial-up test system.

[0317] The message bearer protocol transmitted between the UE and the UPF can be any standard protocol type, such as Hypertext Transfer Protocol Secure (HTTPS), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Quick UDP Internet Connection (QUIC), or UDP, without any limitation.

[0318] Authentication request #1 corresponds to the fourth authentication request mentioned above. Please refer to the relevant description of the fourth authentication request above, which will not be repeated here.

[0319] S606, the UPF sends authentication response #1 to the UE. Correspondingly, the UE receives authentication response #1 from the UPF.

[0320] Among them, authentication response #1 is the response message corresponding to authentication request #1, which is used to notify the first automatic dialing test system that the security authentication was successful.

[0321] UPF determines whether the UE is a test UE in the automatic dialing test system that meets the monitoring conditions based on the UE ID (IMSI / MSISN, etc.) in the received authentication request #1. If so, it checks whether the target address in authentication request #1 is the same as the address of the dialing test server. If so, it parses the authentication account and authentication key carried in authentication request #1 according to the format and matches them with the authentication account and authentication key stored locally. If they match, the security authentication of the UE is completed, and authentication response #1 is sent to the UE to indicate that the security authentication of the first automatic dialing test system corresponding to the test UE is successful.

[0322] If security authentication fails, authentication response #1 indicates that the first automatic dialing test system has failed security authentication.

[0323] Authentication response #1 corresponds to the fourth authentication response mentioned above. Please refer to the relevant description of the fourth authentication response above, which will not be repeated here.

[0324] Optionally, after successfully authenticating the first automatic dialing test system, the UPF can also notify the NWDAF of the successful authentication. For example, the UPF sends an automatic dialing test authentication notification (Nupf_AutoLabel_CertificationNotify) message to the NWDAF, which carries the authenticated dialing test UE ID (IMSI / MSISN, etc.) to notify the first automatic dialing test system of successful authentication.

[0325] S607, the UE sends authentication response #1 to the automatic dialing test control PC. Correspondingly, the automatic dialing test control PC receives authentication response #1 from the UE.

[0326] After receiving the authentication response in the first automatic dialing test system, the UE, upon learning that the security authentication was successful, can further feed back the authentication response #1 to the automatic dialing test control PC for subsequent APP dialing tests, data collection and reporting, and other tasks.

[0327] The above S601 to S607 describe the authentication process of UPF as the dialing test server communicating with the automatic dialing test system on the end side. Thus, after NWDAF learns that the first automatic dialing test system has successfully authenticated, it can determine that a secure channel can be established between UPF and the first automatic dialing test system on the end side. Based on this secure channel, the first task related to the inference or training of the first model deployed on the live network can be completed. The following describes the service recognition rate obtained by the inference of the first model with the first task as the starting point.

[0328] S608, NWDAF configuration startup verification of the business recognition rate obtained from the first model inference.

[0329] After the secure channel is established, NWDAF is configured to automatically or immediately start the verification task of the intelligent SA scenario on the video APP at regular intervals or periodically, that is, to verify the recognition rate of the first model in recognizing the video APP service accessed by the user.

[0330] S609 and NWDAF send an automatic test task allocation message to UPF. Correspondingly, UPF receives the automatic test task allocation message from NWDAF.

[0331] The automatic testing task allocation message (Nupf_AutoLabel_TaskDispatch) is used to issue a task to verify the business recognition rate obtained by the first model inference, or to request the execution of a task to verify the business recognition rate obtained by the first model inference.

[0332] NWDAF can decompose the task of verifying the service recognition rate obtained by the first model inference into the following sub-tasks: The first automatic dialing test system on the terminal side completes the dialing test of the video APP, data cleaning and traffic statistics, data annotation (sent to the first automatic dialing test system on the terminal side via the user plane channel of UPF), annotation data reporting, UPF starts the task of collecting and reporting the user's video service access information, UPF starts the monitoring capability of the annotation data reported by the first automatic dialing test system on the terminal side, and supports reporting to NWDAF.

[0333] For example, information elements in an automatic dialing task allocation message may include:

[0334] --NF ID (Authentication Element Identifier)

[0335] --- Target of Certification (User filtering conditions that need to be monitored and authenticated, such as all users, users who meet specified conditions, such as the ID of the UE being tested in the first automatic dialing test system)

[0336] --APP ID (The identifier of the test APP or the test script, such as the video APP ID)

[0337] --TaskList(s) (e.g., APP testing, packet capture, data cleaning and traffic statistics, data annotation, and annotated data reporting)

[0338] --(Optional) Duration(s) (Duration of test task execution)

[0339] Therefore, NWDAF can break down the task of verifying the business recognition rate obtained from the first model inference into multiple sub-tasks and send them to UPF through automatic test task allocation messages.

[0340] The automatic testing task allocation message corresponds to the first task request or the second task request mentioned above. For details, please refer to the relevant descriptions of the first task request or the second task request in the above embodiments, which will not be repeated here.

[0341] S610 and UPF send an automatic dialing test task start message to the UE. Correspondingly, the UE receives the automatic dialing test task start message from the UPF.

[0342] The automatic testing task start message instructs the first automatic testing system to initiate tasks such as testing the video app, data cleaning and traffic statistics, data annotation, and reporting of annotated data. The automatic testing task start message may include the identifier of the video app being tested or the identifier of the testing script. Optionally, the automatic testing task start message may also include information such as the type of service being tested (live streaming, watching live streaming, short videos, news, long videos, etc.) and the duration of the test, without limitation.

[0343] UPF initiates a monitoring task for service access and labeled data reporting to the first automatic testing system based on the automatic testing task allocation message, and sends an automatic testing task start message to the UE in the first automatic testing system to trigger the first automatic testing system to start testing the video APP and report labeled data.

[0344] The automatic dialing test task start message corresponds to the third task request mentioned above. For details, please refer to the relevant description of the third task request in the above embodiments, which will not be repeated here.

[0345] S611, the UE sends an automatic testing task start message to the automatic testing control PC. Correspondingly, the automatic testing control PC receives the automatic testing task start message from the UE.

[0346] After receiving the automatic dialing test task start message, the UE can send the automatic dialing test task start message back to the automatic dialing test control PC, so that the automatic dialing test control PC can control the UE to perform APP dialing test and data collection.

[0347] Optionally, the UE may also send a response message corresponding to the automatic dialing test task start message to the UPF to instruct the first automatic dialing test system to accept the assigned task or start the assigned task.

[0348] Optionally, after receiving the response message from the first automatic dialing test system confirming the start of the task distribution, the UPF can further report the task's effectiveness to the NWDAF, such as through the response message corresponding to the automatic dialing test task allocation message or other notification messages.

[0349] The automatic testing task start message corresponds to the third task request in the above embodiments, and will not be described in detail here.

[0350] S612, the automatic dialing control PC sends dialing and data acquisition commands to the UE. Correspondingly, the UE receives the dialing and data acquisition commands from the automatic dialing control PC.

[0351] The dial-up and acquisition commands are used to instruct the dial-up video APP and the acquisition of the video APP's business flow messages. The dial-up and acquisition commands may include the identifier of the video APP or the dial-up script identifier.

[0352] After receiving the automatic test task start message, the automatic test control PC can send test and data collection commands to the UE via USB connection based on the task. This will control the UE to test the video APP identifier or the video APP corresponding to the test script identifier, initiate the video service access process, and start the data collection function.

[0353] S613, UE triggers the business access process to the video APP based on the dial-up test and collection instructions.

[0354] Based on the dial-up test and data acquisition commands, the UE initiates the service access script for the video APP, and the user initiates the service access process for the video service to the video APP server.

[0355] S614. The UE sends the collected service flow messages to the automatic dialing test control PC. Correspondingly, the automatic dialing test control PC receives the collected service flow messages from the UE.

[0356] The UE collects the service flow packets accessed by the user during the service access process of the video service, and sends the collected service flow packets to the automatic dial-up control PC after the service access is completed.

[0357] S615, Automatic Diagnostic Control PC obtains labeled data based on the collected service flow messages.

[0358] For example, after the automatic dial-up control PC performs data cleaning and traffic statistics on the service flow packets fed back by the dial-up UE, it can use the identifier of the video APP to label the service flow based on preset rules (such as selecting the service flow with the TOP traffic, the service flow with the same TOP flow server, domain name or server address range), thereby obtaining labeled data such as the service flow five-tuple information and traffic statistics with the identifier of video APP.

[0359] For a detailed description of the labeled data, please refer to the relevant description of the labeled data in the above embodiments, which will not be repeated here.

[0360] S616. The Automatic Dialing Control PC sends annotation data and annotation data reporting instructions to the UE. Correspondingly, the UE receives the annotation data and annotation data reporting instructions from the Automatic Dialing Control PC.

[0361] The annotation data reporting instruction is used to instruct the reporting of annotation data to the UPF.

[0362] After the automatic dialing control PC obtains the annotation data, it sends the annotation data and annotation data reporting instruction to the UE, instructing the UE to report the annotation data to the UPF.

[0363] S617, the UE sends a labeling data reporting request to the UPF. Correspondingly, the UPF receives the labeling data reporting request from the UE.

[0364] Among them, the annotation data reporting request is used to request the reporting of annotation data, and the annotation data reporting request includes the annotation data.

[0365] The UE sends the annotation data to the UPF in the annotation data reporting request, according to the annotation data reporting instruction sent by the automatic dialing control PC.

[0366] S618, UPF sends annotation data to NWDAF. Correspondingly, NWDAF receives annotation data from UPF.

[0367] UPF parses the annotation data in the annotation data reporting request and encapsulates the annotation data into an extended message of the Nupf interface and reports it to NWDAF.

[0368] For example, the labeled data is encapsulated in an Automatic Data Exposure (Nupf_AutoLabel_DataExposure) message, used to report the labeled data to NWDAF. Information elements in the Automatic Data Exposure message may include:

[0369] --NF ID (e.g., the identifier of the first automatic dialing test system)

[0370] --SDF(s) (stream description information, also known as quintuple information)

[0371] --APP ID(s) (Application identifier for business processes, such as the identifier of a video app for testing)

[0372] --Uplink / downline traffic statistics (uplink / downline traffic statistics for service flows accessed during dial-up testing)

[0373] --Start / Stop time (Start and end times of business testing)

[0374] The labeled data consists of APP ID(s), SDF(s) associated with APP ID(s), and uplink / downlink traffic statistics.

[0375] S619 and UPF collect business flow messages and obtain the first data during the business access process.

[0376] For example, during the UE's call test of the video APP and the service access process of the video APP through the user plane, the UPF collects video service flow packets, counts the traffic, total traffic, and five-tuple information of the accessed service flow, and identifies which APP the current accessed service flow corresponds to based on the first model deployed on the UPF, obtains the APP identification result corresponding to the service flow, and reports the APP identification result, traffic, total traffic, and five-tuple information corresponding to the service flow as the first data to the NWDAF.

[0377] S620 and UPF send the first data to NWDAF. Correspondingly, NWDAF receives the first data from UPF.

[0378] For example, UPF can also encapsulate the acquired first data into an extended message of the Nupf interface and report it to NWDAF.

[0379] S621 and NWDAF compare the labeled data with the first data to determine whether the business recognition rate of the first model meets expectations.

[0380] NWDAF compares the first data reported by UPF with the labeled data reported by the first automatic testing system to confirm whether the business recognition rate of the first model meets expectations. For example, if the recognition rate of the first model for video apps is lower than the recognition rate threshold, an alarm is triggered, and subsequent sample collection and model fine-tuning tasks are initiated. The execution process is similar to the business recognition rate task of the recognition model shown in S608 to S621 above, and will not be described in detail here.

[0381] exist Figure 6 In the communication method shown, NWDAF can use UPF as the dial-up test server to communicate with the end-side automatic dial-up test system, completing the end-to-end availability verification process for intelligent SA scenarios with accurate application type identification. It should be understood that, based on... Figure 6 The flowchart shown can also be used to complete tasks such as sample collection and model fine-tuning.

[0382] In other words, the data marked by the end-side automatic dialing test system is reported to the UPF through the user plane path, and the authentication process is completed through the data plane transmission channel between the UPF and the end-side system. The authentication account / key for the end-side system on the UPF can be issued by the NWDAF or configured locally. After the UPF completes the authentication of the end-side system, the task issuance and data interaction between the end-side automatic dialing test system and the NWDAF can be realized by combining the extended interface capabilities of the UPF and the NWDAF.

[0383] For example, taking a video app as the first application, a sample set for generating the service recognition rate of the first model (or a sample collection task for training the service recognition rate of the model) as the first task, a test server as the first device, an NWDAF as the data analysis network element, and a UE as the first automatic test system, the following example is used: Figure 7 A flowchart illustrating a communication method provided in an embodiment of this application is shown. Figure 7 As shown, the communication method includes:

[0384] S701 and NWDAF are used to monitor the authentication information of the test server.

[0385] The authentication information of the test server may include the address of the test server, the authentication account of the test server, and the authentication key.

[0386] S702, The dial-up test server is configured with the authentication information of the dial-up test server certified by NWDAF, and the first authentication information of the automatic dial-up test system to be monitored.

[0387] The first authentication information may include the address of the dial-up test server, the identifier of the automatic dial-up test system to be monitored, such as the IMSI or MSISDN of the UE in the automatic dial-up test system to be monitored, the authentication account and the authentication key.

[0388] S703, the test server sends authentication request #1 to the NWDAF. Correspondingly, the NWDAF receives authentication request #1 from the test server.

[0389] Specifically, authentication request #1 is used to request NWDAF to perform security authentication on the test server. Authentication request #1 includes the authentication information of the test server. For example, authentication request #1 is Nnwdaf_AutoLabel_CertificationRequest, which may include:

[0390] --NF ID (The identifier of the test server requesting authentication, such as the IP address of the test server)

[0391] --set of)Account Info(s) (List of verified accounts and keys)

[0392] --(Optional) Timeout (Timeout duration for each authentication)

[0393] Optionally, the first authentication information of the automatic dial-up testing system to be monitored can be sent to the dial-up testing server by NWDAF through a customized message, similar to the automatic dial-up testing authentication request in S602 above.

[0394] S704, NWDAF sends authentication response #1 to the test server. Correspondingly, the test server receives authentication response #1 from NWDAF.

[0395] Among them, authentication response #1 is used to notify the test server that the security authentication was successful. Authentication response #1 is the response message corresponding to authentication request #1.

[0396] NWDAF can match the authentication information of the local test server with the authentication information of the test server in authentication request #1. If the match is successful, the test server is successfully authenticated and authentication response #1 is sent to the test server.

[0397] If a match is not found, the security authentication of the test server will fail. In this case, authentication response #1 is used to notify the test server of the security authentication failure.

[0398] Authentication request #1 and authentication response #1 correspond to the second authentication request and the second authentication response in Design 2 above, respectively. Please refer to the relevant descriptions of the second authentication request and the second authentication response above, which will not be repeated here.

[0399] S705, the UE sends authentication request #2 to the test server. Correspondingly, the test server receives authentication request #2 from the UE.

[0400] Among them, authentication request #2 is used to request security authentication of the first automatic dialing test system, and authentication request #2 includes the authentication information of the first automatic dialing test system.

[0401] After the first automatic dial-up testing system is started, the UE can access the testing server through normal user plane data and initiate an authentication process by sending authentication request #2. Authentication request #2 carries information such as the testing server address, UE ID, authentication account, and authentication key used for authentication by the first automatic dial-up testing system.

[0402] S706. The test server sends authentication response #2 to the UE. Correspondingly, the UE receives authentication response #2 from the test server.

[0403] Among them, authentication response #2 is the response message corresponding to authentication request #2, which is used to notify the first automatic dialing test system that the security authentication was successful.

[0404] The testing server initiates monitoring and authentication of the designated automatic testing system that meets the user filtering conditions. Upon receiving authentication request #2, it determines whether the UE is a testing UE in the automatic testing system that meets the monitoring conditions based on the UE ID (IMSI / MSISN, etc.) in the received authentication request #2. If so, it checks whether the target address in authentication request #2 is the same as the address of the testing server. If so, it parses the authentication account and authentication key carried in authentication request #2 according to the format and matches them with the authentication account and authentication key stored locally. If they match, the security authentication of the UE is completed, and authentication response #2 is sent to the UE to indicate that the security authentication of the first automatic testing system corresponding to the testing UE is successful.

[0405] If security authentication fails, authentication response #2 indicates that the first automatic dialing test system has failed to authenticate.

[0406] Authentication request #2 and authentication response #2 correspond to the fourth authentication request and the fourth authentication response mentioned above, respectively. Please refer to the relevant descriptions of the fourth authentication request and the fourth authentication response mentioned above, which will not be repeated here.

[0407] Optionally, after successfully authenticating the first automatic dial-testing system, the dial-testing server can also notify the NWDAF of the successful authentication. For example, the dial-testing server sends an automatic dial-testing authentication notification message to the NWDAF, which carries the authenticated dial-testing UE ID (IMSI / MSISN, etc.) to notify the first automatic dial-testing system of successful authentication.

[0408] The above S701 to S706 describe the authentication process for communication between the independently set up dial-up test server in the network and the automatic dial-up test system on the end side. Thus, after NWDAF learns that the first automatic dial-up test system has successfully authenticated, it determines that a secure channel can be established between the dial-up test server and the first automatic dial-up test system on the end side. Based on this secure channel, the first task related to the inference or training of the first model deployed on the live network can be completed. The following describes the first task as generating a sample set for training the service recognition rate of the first model in detail.

[0409] The S707 and NWDAF configuration starts generating a sample set of business recognition rates for training the first model.

[0410] After the secure channel is established, NWDAF is configured to automatically start or immediately start the task of generating a sample set for training business recognition rate in the intelligent SA scenario at regular intervals or periodically.

[0411] S708 and NWDAF send Automatic Test Task Allocation Message #1 to the Test Server. Correspondingly, the Test Server receives Automatic Test Task Allocation Message #1 from NWDAF.

[0412] Among them, the automatic dialing test task allocation message #1 is used to issue the first automatic dialing test system to execute tasks related to the sample set for generating the business recognition rate of the first model.

[0413] NWDAF can decompose the task of generating a sample set for training the first model's business recognition rate into sub-tasks completed by the first automatic testing system, such as video APP testing, data collection, data cleaning and traffic statistics, data annotation (sent to the end-side automatic testing system via the user plane channel of the testing server), and annotation data reporting; sub-tasks completed by the UPF, such as the UPF initiating the collection and reporting of the first data of the video APP; and sub-tasks completed by the testing server, such as the testing server initiating the monitoring capability of the annotation data reported by the end-side first automatic testing system and supporting reporting to NWDAF.

[0414] Therefore, the automatic dial-up test task allocation message #1 includes the sub-tasks completed by the first automatic dial-up test system and the sub-tasks completed by the dial-up test server. An example description of the automatic dial-up test task allocation message #1 can be found in the relevant description in S609 above, and will not be repeated here.

[0415] The automatic testing task allocation message #1 corresponds to the second task request in the above embodiment, and will not be described in detail.

[0416] S709, NWDAF sends Automatic Diagnostic Task Assignment Message #2 to UPF. Correspondingly, UPF receives Automatic Diagnostic Task Assignment Message #2 from NWDAF.

[0417] Among them, the automatic dialing test task allocation message #2 includes the sub-task completed by the UPF, namely, the UPF starts to collect and report the first data of the video APP. The first data is the sample feature data collected by the UPF from the user access video service flow for service identification.

[0418] UPF initiates the business flow collection and reporting task for the video APP of the first automatic dial-up test system according to the automatic dial-up test task allocation message #2, and obtains the first data.

[0419] The automatic testing task allocation message #2 corresponds to the first task request in the above embodiment, and will not be described in detail.

[0420] S710. The test server sends an automatic test task start message to the UE. Correspondingly, the UE receives the automatic test task start message from the test server.

[0421] For a detailed description of S710, please refer to the relevant description in S610 above, which will not be repeated here. The automatic dialing test task start message corresponds to the third task request in the above embodiment, which will not be repeated here.

[0422] Optionally, the UE may also send a response message corresponding to the automatic dialing task start message to the dialing test server to instruct the first automatic dialing test system to accept the assigned task or start the assigned task.

[0423] S711, the UE triggers the service access process to the video APP based on the automatic dial-up test task start message.

[0424] According to the automatic dial-up test task start message, the UE starts the service access script for the video APP, and initiates the service access process for the video service through the user to the video APP server.

[0425] S712, UE obtains labeled data based on the collected service flow messages.

[0426] During the video service access process, the UE collects the service flow packets accessed by the user. After the service access is completed, the collected service flow packets are cleaned, traffic statistics are performed, and annotation is performed to obtain the annotation data. The annotation data includes the service flow five-tuple information with the tag "video APP", traffic statistics data, etc.

[0427] S713, the UE sends a labeling data reporting request to the test server. Correspondingly, the UPF receives the labeling data reporting request from the UE.

[0428] Among them, the annotation data reporting request is used to request the reporting of annotation data, and the annotation data reporting request includes the annotation data.

[0429] S714. The test server sends annotation data to the NWDAF. Correspondingly, the NWDAF receives annotation data from the test server.

[0430] S715 and UPF collect business flow messages and obtain the first data during the business access process.

[0431] For example, when a UE dials and tests a video app, during the service access process of accessing the video app through the user plane, the UPF collects video service flow packets and extracts service flow feature data, such as five-tuple information, first data of the service flow, and service flow load characteristics, from the collected video service flow packets, thereby obtaining the first data.

[0432] S716, UPF sends the first data to NWDAF. Correspondingly, NWDAF receives the first data from UPF.

[0433] For example, the UPF transmits the first data to the NWDAF via Nupf_AutoLabel_DataExposure, which may include:

[0434] --NF ID (e.g., the identifier of the first automatic dialing test system)

[0435] --SDF(s) (stream description information, also known as quintuple information)

[0436] --APP ID(s) (Application identifier for business processes, such as the identifier of a video app for testing)

[0437] --Statistical characteristics of the business flow (such as the packet length sequence of the first N packets, protocol type, uplink / downlink traffic ratio, mean, variance, etc.)

[0438] --Service Flow Load Characteristics

[0439] Among them, SDF(s), business flow statistical characteristics, and business flow load characteristics constitute the first data.

[0440] S717 and NWDAF generate a sample set for training the business recognition rate of the first model based on the labeled data and the first data.

[0441] NWDAF associates the first data reported by UPF, which includes the business flow quintuple and business flow features, with the labeled business access information reported by the first automatic dialing system, i.e., the labeled data, with the labels and sample features to generate sample data containing sample features. Then, it generates training and validation sets based on the sample data to train / fine-tune the business recognition capability of the first model and improve the recognition rate of intelligent business perception.

[0442] exist Figure 7 In the communication method shown, NWADF communicates with the automatic testing system on the device side through an independent testing server to complete the sample data acquisition process for the specified APP. It should be understood that, based on... Figure 7 The process shown can also complete the above-mentioned tasks such as model usability verification, model training / fine-tuning.

[0443] In other words, the data marked by the end-side automatic dial-up testing system is reported to the dial-up testing server through the user plane path. The automatic dial-up testing system and the dial-up testing server complete the authentication process. The authentication account / key of the end-side system can be issued by NWDAF or configured locally. After the dial-up testing server completes the authentication of the end-side system, the task distribution and data interaction between the end-side automatic dial-up testing system and NWDAF can be realized by combining the extended interface capabilities of the dial-up testing server and NWDAF.

[0444] For example, taking a video app as the first application, the first task as generating a sample set for service quality assessment to train a first model (or a sample collection task for service quality assessment to train the model), the first device as a security bastion host, the data analysis network element as NWDAF, and the first automatic dialing test system including a UE and an automatic dialing test control PC as an example, Figure 8 A flowchart illustrating a communication method provided in an embodiment of this application is shown. Figure 8 As shown, the communication method includes:

[0445] S801 and NWDAF configure the first authentication information of the automatic dialing test system to be monitored.

[0446] The first authentication information may include the address of the automatic dialing control PC, the address of the dialing server, the identifier of the automatic dialing system to be monitored, such as the IMSI or MSISDN of the UE in the automatic dialing system to be monitored, the authentication account and the authentication key.

[0447] S802, the automatic dial-test control PC sends authentication request #1 to the security bastion host. Correspondingly, the security bastion host receives authentication request #1 from the automatic dial-test control PC.

[0448] The authentication request #1 is used to request security authentication for the first automatic dialing test system. The authentication request #1 includes the authentication information of the first automatic dialing test system, which may include information for security bastion host authentication and information for NWDAF authentication.

[0449] The automatic dialing control PC can send authentication request #1 to the security bastion host via a wired connection. The security bastion host determines whether the address of the automatic dialing control PC is the address to be monitored, and whether the authentication account and authentication key are correct, based on the authentication information of the first automatic dialing system in the authentication request #1. If so, the authentication request #1 is forwarded to NWDAF.

[0450] S803, the security bastion host sends authentication request #1 to the NWDAF. Correspondingly, the NWDAF receives authentication request #1 from the security bastion host.

[0451] S804, NWDAF sends authentication response #1 to the security bastion host. Correspondingly, the security bastion host receives authentication response #1 from NWDAF.

[0452] NWDAF determines whether the UE is a test UE in the automatic dialing test system that meets the monitoring conditions based on the UE ID (IMSI / MSISN, etc.) in the received authentication request #1. If so, it checks whether the target address in authentication request #1 is the same as the address of the dialing test server. If so, it parses the authentication account and authentication key carried in authentication request #1 according to the format and matches them with the authentication account and authentication key stored locally. If they match, the security authentication of the UE is completed, and authentication response #1 is sent to the UE to indicate that the security authentication of the first automatic dialing test system corresponding to the test UE is successful.

[0453] If security authentication fails, authentication response #1 indicates that the first automatic dialing test system has failed security authentication.

[0454] S805, the security bastion host sends authentication response #1 to the automatic dial-test control PC. Correspondingly, the automatic dial-test control PC receives authentication response #1 from the security bastion host.

[0455] The above S801 to S805 describe the process by which the automatic dialing control PC in the end-side automatic dialing test system performs security authentication with the NWDAF through the security bastion host. After the NWDAF learns that the first automatic dialing test system has successfully authenticated, it determines that a secure channel can be established between the NWDAF and the first automatic dialing test system on the end-side through the security bastion host. Thus, it can complete the first task related to the inference or training of the first model deployed on the live network based on this secure channel. The following describes the first task in detail using the sample set for generating the service quality assessment of the first model.

[0456] S806, NWDAF configuration starts to generate a sample set for the business quality assessment of the first model.

[0457] After the secure channel is established, NWDAF is configured to automatically start or immediately start the task of generating a sample set for business quality assessment in the intelligent QoE scenario at regular intervals or periodically.

[0458] S807 and NWDAF send an automatic dial-test task start message to the security bastion host. Correspondingly, the security bastion host receives the automatic dial-test task start message from NWDAF.

[0459] The automatic testing task start message is used to instruct the first automatic testing system to start testing, data cleaning and traffic statistics, data labeling, and reporting of labeled data for the video APP.

[0460] NWDAF decomposes the task of generating the sample set for the business quality assessment of the first model into sub-tasks for the first automatic testing system, such as testing of the video APP, data cleaning and traffic statistics, data annotation (sent to the first automatic testing system on the end side via the security bastion host channel), and reporting of annotated data; sub-tasks for the UPF, such as the UPF starting to collect and report the first data of the video APP; and sub-tasks for the NWDAF, such as the NWDAF starting to acquire the annotated data reported by the first automatic testing system on the end side, and supports the AI ​​model training and fine-tuning tasks based on the sample set that meets the conditions.

[0461] The automatic testing task start message corresponds to the second task request in the above embodiment, and will not be described in detail here.

[0462] S808 and NWDAF send Automatic Diagnostic Task Allocation Message #1 to UPF. Correspondingly, UPF receives Automatic Diagnostic Task Allocation Message #1 from NWDAF.

[0463] Among them, the automatic dialing test task allocation message #1 includes the sub-task completed by the UPF, namely, the UPF starts to collect and report the first data of the video APP. The first data is the sample feature data of the user access video service flow collected by the UPF for QoE evaluation.

[0464] UPF initiates the business flow collection and reporting task for the video APP of the first automatic dial-up test system according to the automatic dial-up test task allocation message #1, and obtains the first data.

[0465] The automatic testing task allocation message #1 corresponds to the first task request in the above embodiment, and will not be described in detail.

[0466] S809. The security bastion host sends an automatic dialing test task start message to the UE. Correspondingly, the UE receives the automatic dialing test task start message from the security bastion host.

[0467] S810: The UE sends an automatic testing task start message to the automatic testing control PC. Correspondingly, the automatic testing control PC receives the automatic testing task start message from the UE.

[0468] After receiving the automatic dialing test task start message, the UE can send the automatic dialing test task start message back to the automatic dialing test control PC, so that the automatic dialing test control PC can control the UE to perform APP dialing test and data collection.

[0469] Optionally, the UE can also send a response message corresponding to the automatic dialing test task start message to the security bastion host, instructing the first automatic dialing test system to accept or start the assigned task, and then the security bastion host can provide feedback to the NWDAF.

[0470] S811, the automatic dialing control PC sends dialing and data acquisition commands to the UE. Correspondingly, the UE receives the dialing and data acquisition commands from the automatic dialing control PC.

[0471] S812 and UE trigger the business access process to the video APP based on the dial-up test and collection instructions.

[0472] S813, the UE sends the collected service flow messages to the automatic dialing test control PC. Correspondingly, the automatic dialing test control PC receives the collected service flow messages from the UE.

[0473] S814, Automatic Diagnostic Control PC obtains labeled data based on the collected service flow messages.

[0474] S815, the Automatic Dialing Control PC sends annotation data and annotation data reporting instructions to the UE. Correspondingly, the UE receives the annotation data and annotation data reporting instructions from the Automatic Dialing Control PC.

[0475] For detailed descriptions of S811 to S815, please refer to the relevant descriptions of S612 to S615 above, which will not be repeated here. The difference from S612 to S615 is that the labeled data uses QoE or KQI scores as tags. The labeled business flow access data is related to the QoE or KQI score. That is, the labeled data includes business access data such as the QoE or KQI score as a tag, five-tuple information, APP identifier, and QoE or KQI information. QoE or KQI information includes parameters such as resolution, bitrate, first frame loading time, first frame loading time scaling rate, stuttering duration, number of stutters, video playback success rate, startup download speed, media data transmission round-trip latency, media data transmission packet loss rate, and download speed.

[0476] S816, the UE sends a labeling data reporting request to the security bastion host. Correspondingly, the security bastion host receives the labeling data reporting request from the UE.

[0477] Among them, the annotation data reporting request is used to request the reporting of annotation data, and the annotation data reporting request includes the annotation data.

[0478] S817, the security bastion host sends annotation data to the NWDAF. Correspondingly, the NWDAF receives annotation data from the security bastion host.

[0479] S818 and UPF collect business flow messages and obtain the first data during the business access process.

[0480] For example, during the service access process of a UE dialing a video app and accessing the video app service through the user plane, the UPF collects video service flow packets, extracts sample feature data for QoE evaluation, and generates first data. The first data includes the five-tuple information of the service flow and the feature sample data of QoE evaluation, such as the first data of the service flow and the load characteristics of the service flow.

[0481] S819, UPF sends the first data to NWDAF. Correspondingly, NWDAF receives the first data from UPF.

[0482] For example, the UPF transmits the first data to the NWDAF via Nupf_AutoLabel_DataExposure, which may include:

[0483] --NF ID (e.g., the identifier of the first automatic dialing test system)

[0484] --SDF(s) (stream description information, also known as quintuple information)

[0485] --APP ID(s) (Application identifier for business processes, such as the identifier of a video app for testing)

[0486] --Statistical characteristics of the business flow (such as the packet length sequence of the first N packets, protocol type, uplink / downlink traffic ratio, mean, variance, etc.)

[0487] --Service Flow Load Characteristics

[0488] Among them, SDF(s), business flow statistical characteristics, and business flow load characteristics constitute the first data.

[0489] S820 and NWDAF generate a sample set for business quality assessment to train the first model based on the labeled data and the first data.

[0490] NWDAF associates the business flow quintuple and QoE / KQI evaluation feature samples reported by UPF (i.e., the first data) with the labeled business access information and QoE / KQI values ​​reported by the first automatic dialing system (i.e., the labeled data) with the labels and sample features to generate sample data containing the sample features. Based on this sample data, training and validation sets are generated to train / fine-tune the AI ​​model and improve the accuracy of intelligent experience evaluation indicators.

[0491] Figure 8 In the communication method shown, NWADF communicates with the automatic probing system on the endpoint via a security bastion host to complete the sample data collection process for the specified application, in order to train or fine-tune the model. It should be understood that, based on... Figure 7 The process shown can also complete the above-mentioned model usability verification task process.

[0492] In other words, the data marked by the end-side automatic dialing test system is relayed to the NWDAF device through a security bastion host with security authentication. The authentication account / key of the security bastion host for the end-side system can be issued by the NWDAF or configured locally. After the security bastion host completes the authentication of the end-side system, it can realize the task distribution and data interaction between the end-side automatic dialing test system and the NWDAF through the relay channel established between the automatic dialing test system and the NWDAF.

[0493] Figure 7 and Figure 8 The results show that NWDAF supports comparison with the intelligent service perception and experience evaluation system within the 3GPP system based on labeled sample data, realizing training and sample set generation in intelligent service perception and intelligent service evaluation scenarios.

[0494] In this embodiment, NWDAF supports the development of new functions related to all solutions involving APP testing, annotation, model evaluation and verification. The automatic testing system supports tasks such as testing, data collection and annotation, and traffic statistics issued by NWDAF, driving the testing terminal device to trigger corresponding APP business access, and supports performing data cleaning and annotation tasks on the terminal side, and then reporting the relevant annotation data to NWDAF through different paths. Devices such as UPF, testing data collection server, and security bastion host support the construction of a secure channel between the automatic testing system and NWDAF, and realize functions such as business access information statistics and reporting, sample feature collection, and annotation data acquisition.

[0495] It should be understood that, in addition to the end-side automatic dialing test system mentioned in the above embodiments, which uses three channels—UPF, dialing test server, and security bastion host—to realize the authentication, task distribution, and data reporting processes and interfaces with NWDAF, there may be other NFs and data channels that can realize the authentication, task distribution, and data reporting processes and solutions between the end-side automatic dialing test system and NWDAF, and there are no limitations on this.

[0496] In addition to the intelligent SA and service experience QoE evaluation scenarios mentioned in the above embodiments, which can be implemented by the local network collaborative automated dialing test system to achieve functional verification, tag data collection and reporting, and model training and fine-tuning tasks, there may be other intelligent scenarios that can be implemented by the local network collaborative automated dialing test system to achieve functional verification, tag data collection and reporting, and model training and fine-tuning tasks. There are no limitations on these scenarios.

[0497] In addition to the task distribution, tag data reporting, statistics and feature reporting interfaces and processes extended by the Nupf, Nnwdaf and Nxxx interfaces mentioned in the above embodiments, there may be other interfaces and processes that can realize end-to-end intelligent scenarios for task distribution, tag data reporting, statistics and feature reporting, and there are no limitations on these.

[0498] In the above embodiments, the methods and / or steps implemented by the data analysis network element can also be implemented by components (e.g., processors, chips, chip systems, circuits, logic modules, or software) that can be used in the data analysis network element; in the above embodiments, the methods and / or steps implemented by the first device can also be implemented by components (e.g., processors, chips, chip systems, circuits, logic modules, or software) that can be used in the first device; in the above embodiments, the methods and / or steps implemented by the first automatic dialing system can also be implemented by components (e.g., processors, chips, chip systems, circuits, logic modules, or software) that can be used in the first automatic dialing system.

[0499] The foregoing mainly describes the solutions provided in this application. Accordingly, this application also provides a communication device for implementing various methods in the above method embodiments. This communication device can be a data analysis network element in the above method embodiments, or a device containing a data analysis network element, or a component that can be used in a data analysis network element, such as a chip or chip system. Alternatively, the communication device can be the first device in the above method embodiments, or a device containing the first device, or a component that can be used in a second network element, such as a chip or chip system. Alternatively, the communication device can be the first automatic dialing test system in the above method embodiments, or a device containing the first automatic dialing test system, or a component that can be used in the first automatic dialing test system, such as a chip or chip system.

[0500] In some embodiments, the communication device includes hardware structures and / or software modules corresponding to the execution of each function in order to achieve the above-described functions. Those skilled in the art will readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0501] This application embodiment can divide the communication device into functional modules according to the above method embodiment. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0502] Taking a communication device as an example, which is the data analysis network element, the first device, or the first automatic dialing system in the above method embodiments, Figure 9 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application. Figure 9 As shown, the communication device 900 includes a processing module 901 and a transceiver module 902. The processing module 901 is used to execute the processing functions of the data analysis network element, the first device, or the first automatic dialing test system in the above method embodiments. The transceiver module 902 is used to execute the communication functions of the data analysis network element, the first device, or the first automatic dialing test system in the above method embodiments.

[0503] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0504] In one possible design, in this embodiment of the application, the transceiver module 902 may include a receiving module and a sending module. Figure 9 (Not shown in the diagram). The transmitting module and receiving module are used to implement the transmitting and receiving functions of the communication device 900, respectively.

[0505] In one possible design, the communication device 900 may further include a storage module. Figure 9 (Not shown in the image), this storage module stores programs or instructions. When the processing module 901 executes the program or instructions, it enables the communication device 900 to perform... Figures 5-8The function of the data analysis network element, the first device, or the first automatic dialing system in any of the methods shown.

[0506] In some embodiments, the processing module 901 involved in the communication device 900 may be implemented by a processor or processor-related circuit components, and may be a processor or processing unit; the transceiver module 902 may be implemented by a transceiver or transceiver-related circuit components, and may be a transceiver or transceiver unit.

[0507] For example, Figure 10 This is a schematic diagram of another communication device provided in an embodiment of this application. The communication device can be a data analysis network element, a first device, or a first automatic dialing system as described in the above method embodiments; it can also be a chip (system) or other component or assembly that can be disposed in the data analysis network element, the first device, or the first automatic dialing system. Figure 10 As shown, the communication device 1000 may include a processor 1001. In one possible design, the communication device 1000 may further include a memory 1002 and / or a transceiver 1003. The processor 1001 is coupled to the memory 1002 and the transceiver 1003, for example, they may be connected via a communication bus.

[0508] The following is combined Figure 10 A detailed description of each component of the communication device 1000 is provided below:

[0509] The processor 1001 is the control center of the communication device 1000. It can be a single processor or a collective term for multiple processing elements. For example, the processor 1001 may include one or more CPUs, or it may be an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0510] In one possible design, the processor 1001 can perform various functions of the communication device 1000 by running or executing software programs stored in the memory 1002 and calling data stored in the memory 1002.

[0511] In a specific implementation, as one example, the processor 1001 may include one or more CPUs, for example... Figure 10 CPU0 and CPU1 are shown in the diagram.

[0512] In a specific implementation, as one example, the communication device 1000 may also include multiple processors, for example... Figure 10 The processors 1001 and 1004 are shown. Each of these processors may be a single-core processor or a multi-core processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0513] The memory 1002 is used to store the software program that executes the solution of this application, and is controlled by the processor 1001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0514] In one possible design, the memory 1002 can be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. The memory 1002 can be integrated with the processor 1001 or exist independently, and can be accessed via the interface circuit of the communication device 1000. Figure 10 (Not shown in the image) is coupled to the processor 1001, and this embodiment does not specifically limit this.

[0515] Transceiver 1003 is used for communication with other communication devices. For example, if communication device 1000 is a terminal device, transceiver 1003 can be used for communication with network devices or with another terminal device. As another example, if communication device 1000 is a network device, transceiver 1003 can be used for communication with a terminal device or with another network device.

[0516] In one possible design, transceiver 1003 may include a receiver and a transmitter. Figure 10 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0517] In one possible design, the transceiver 1003 can be integrated with the processor 1001, or it can exist independently and be connected via the interface circuit of the communication device 1000. Figure 10 (Not shown in the image) is coupled to the processor 1001, and this embodiment does not specifically limit this.

[0518] It should be noted that, Figure 10 The structure of the communication device 1000 shown does not constitute a limitation on the communication device. Actual communication devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0519] Furthermore, the technical effects of the communication device 1000 can be referred to the technical effects of the method described in the above method embodiments, and will not be repeated here.

[0520] In another aspect, embodiments of this application also provide a computer program product containing instructions, including computer program code, which, when run on a communication device, enables the communication device to execute the methods described in any of the above embodiments.

[0521] Furthermore, embodiments of this application also provide a computer-readable storage medium. This computer-readable storage medium stores a computer program or instructions that, when executed on a communication device, enable the communication device to perform the methods described in any of the above embodiments.

[0522] In another aspect, embodiments of this application also provide a communication system, including: a data analysis network element, a first device, and a first automatic dialing system for implementing the above method embodiments.

[0523] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video disks (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0524] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0525] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0526] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0527] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0528] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0529] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or an access network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, random access memory (RAM), magnetic disks, or optical disks.

[0530] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, the disclosure, and the appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0531] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A communication method, characterized in that, The method is applied to a data analysis network element. When a secure channel is established between the first device and the first automatic dialing system, the method includes: Send a first task request to the user plane network element. The first task request is used to request the user plane network element to obtain and report the first data of the service flow of the first automatic dialing test system accessing the first application. A second task request is sent to the first device, the second task request being used to request the first device to trigger the first automatic dialing system to acquire and report labeled data of the business flow accessing the first application; wherein, the first data and the labeled data are used to complete a first task related to business inference or training of the first model, the first model being used for business-related inference; The system receives the first data from the user plane network element and the labeled data from the first device.

2. The method according to claim 1, characterized in that, The first automatic dialing test system includes a first terminal device for dialing test of the first application.

3. The method according to claim 1 or 2, characterized in that, The first device triggers the first automatic dialing test system to acquire and report labeled data of the service flow accessing the first application, including: the first device triggers the first automatic dialing test system to start dialing test of the first application, dialing test data packet capture, dialing test data cleaning and traffic statistics, dialing test data collection and labeling, labeled data reporting, and the first device starts monitoring and reporting the labeled data reported by the first automatic dialing test system.

4. The method according to any one of claims 1-3, characterized in that, The first task is to verify whether the business reasoning results of the first model meet expectations, or the first task is to generate a sample set for training the first model.

5. The method according to claim 4, characterized in that, If the first task is to verify whether the business inference result of the first model meets expectations, then the first data is used to compare with the labeled data. The first data includes the business inference result, the business flow access data obtained by the user plane network element from the collected business flow of the first application, and the quintuple information. The business inference result is obtained by the user plane network element using the first model to infer the business flow access data obtained by the user plane network element.

6. The method according to claim 4, characterized in that, If the first task is to generate a sample set for training the first model, then the first data is used to associate with the labeled data. The first data includes the service flow feature data and quintuple information of the service flow of the first application collected by the user plane network element.

7. The method according to any one of claims 4-6, characterized in that, The business reasoning result is either a business identification result or a business quality assessment result, and the sample set is used to train the business identification capability or business quality assessment capability of the first model.

8. The method according to claim 7, characterized in that, If the business reasoning result is the business identification result or the sample set is used to train the business identification capability of the first model, then the labeled data includes the application identifier as a label and the business flow access data and quintuple information associated with the application identifier; If the business reasoning result is the business quality assessment result or the sample set is used to train the business quality assessment capability of the first model, then the labeled data includes the business quality result as a label and the business flow access data and quintuple information associated with the business quality result.

9. The method according to any one of claims 1-8, characterized in that, The establishment of a secure channel between the first device and the first automatic dialing system includes: When the security authentication of the first automatic dialing test system is completed through the first device, a secure channel is established between the first device and the first automatic dialing test system. The secure channel supports the data analysis network element in issuing tasks to the first automatic dialing test system and supports the first automatic dialing test system in reporting the labeled data.

10. The method according to claim 9, characterized in that, The first device is the user plane network element, and the step of completing the security authentication of the first automatic dialing test system through the first device includes: A first authentication request is sent to the user plane network element. The first authentication request is used to request the user plane network element to perform security authentication on the automatic dialing test system to be monitored. The first authentication request includes the first authentication information of the automatic dialing test system to be monitored. The automatic dialing test system to be monitored includes the first automatic dialing test system.

11. The method according to claim 9, characterized in that, The first device is a dial-up testing server corresponding to the first automatic dial-up testing system. The step of completing security authentication of the first automatic dial-up testing system through the first device includes: The system receives a second authentication request from the test server. The second authentication request is used to request the data analysis network element to perform security authentication on the test server. The second authentication request includes the authentication information of the test server. A second authentication response is sent to the test server, which is used to notify the test server that the security authentication was successful.

12. The method according to claim 9, characterized in that, The first device is a security bastion host. The first automatic dialing test system also includes an automatic dialing test control device for controlling the dialing test of the first terminal device of the first application. The security authentication of the first automatic dialing test system through the first device includes: Receive a third authentication request from the security bastion host, the third authentication request being used to request the data analysis network element to perform security authentication on the first automatic dialing test system, the third authentication request including the authentication information of the first automatic dialing test system; A third authentication response is sent to the security bastion host, which is used to notify the first automatic dialing test system that the security authentication was successful.

13. A communication method, characterized in that, Applied to a first device, when the first device completes the establishment of a secure channel between the data analysis network element and the first automatic dialing system, the method includes: The system receives a second task request from the data analysis network element. The second task request is used to request the first device to trigger the first automatic dialing system to acquire and report the labeled data of the service flow accessing the first application. According to the second task request, a third task request is sent to the first automatic dialing test system. The third task request is used to request the first automatic dialing test system to obtain and report the annotation data of the business flow accessing the first application. The annotation data is received from the first automatic testing system. The annotation data is used to complete a first task related to business reasoning or training of the first model. The first model is used for business-related reasoning. The labeled data is sent to the data analysis network element.

14. The method according to claim 13, characterized in that, The first device triggers the first automatic dialing test system to acquire and report labeled data of the service flow accessing the first application, including: the first device triggers the first automatic dialing test system to start dialing test of the first application, dialing test data packet capture, dialing test data cleaning and traffic statistics, dialing test data collection and labeling, labeled data reporting, and the first device starts monitoring and reporting the labeled data reported by the first automatic dialing test system.

15. The method according to claim 13 or 14, characterized in that, The first task is to verify whether the business reasoning results of the first model meet expectations, or the first task is to generate a sample set for training the first model.

16. The method according to claim 15, characterized in that, The business reasoning result is either a business identification result or a business quality assessment result, and the sample set is used to train the business identification capability or business quality assessment capability of the first model.

17. The method according to claim 16, characterized in that, If the business reasoning result is the business identification result or the sample set is used to train the business identification capability of the first model, then the labeled data includes the application identifier as a label and the business flow access data and quintuple information associated with the application identifier; If the business reasoning result is the business quality assessment result or the sample set is used to train the business quality assessment capability of the first model, then the labeled data includes the business quality result as a label and the business flow access data and quintuple information associated with the business quality result.

18. The method according to any one of claims 13-17, characterized in that, The first device is a user plane network element, and the method further includes: Receive a first task request from the data analysis network element. The first task request is used to request the user plane network element to obtain and report first data of the service flow of the first automatic dialing system accessing the first application. The first data is used to complete the first task. The first data is sent to the data analysis network element.

19. The method according to claim 18, characterized in that, If the first task is to verify whether the business inference result of the first model meets expectations, then the first data is used to compare with the labeled data. The first data includes the business inference result, the business flow access data and quintuple information obtained by the user plane network element from the collected business flow of the first application. The business inference result is obtained by the user plane network element using the first model to infer the business flow access data obtained by the user plane network element.

20. The method according to claim 18, characterized in that, If the first task is to generate a sample set for training the first model, then the first data is used to associate with the labeled data. The first data includes service flow feature data and quintuple information obtained by the user plane network element from the service flow of the first application.

21. A communication method, characterized in that, Applied to a first automatic dialing test system, when a secure channel is established between the system and the data analysis network element via a first device, the method includes: Receive a third task request from the first device, the third task request being used to request the first automatic dialing system to acquire and report the annotation data of the service flow accessing the first application; The labeled data is sent to the first device. The labeled data is used to complete a first task related to business reasoning or training of the first model. The first model is used for business-related reasoning.

22. The method according to claim 21, characterized in that, The first automatic testing system acquires and reports labeled data of the business flow accessing the first application, including: the first automatic testing system starts testing the first application, packet capture of testing data, cleaning and traffic statistics of testing data, collection and labeling of testing data, and reporting of labeled data.

23. The method according to claim 21 or 22, characterized in that, The first task is to verify whether the business reasoning results of the first model meet expectations, or the first task is to generate a sample set for training the first model.

24. The method according to claim 23, characterized in that, The business reasoning result is either a business identification result or a business quality assessment result, and the sample set is used to train the business identification capability or business quality assessment capability of the first model.

25. The method according to claim 24, characterized in that, If the business reasoning result is the business identification result or the sample set is used to train the business identification capability of the first model, then the labeled data includes the application identifier as a label and the business flow access data and quintuple information associated with the application identifier; If the business reasoning result is the business quality assessment result or the sample set is used to train the business quality assessment capability of the first model, then the labeled data includes the business quality result as a label and the business flow access data and quintuple information associated with the business quality result.

26. A communication device, characterized in that, Includes modules for performing the methods as described in any one of claims 1-12, 13-20, or 21-25.

27. A communication device, characterized in that, include: processor; The processor is configured to run computer programs or instructions to enable the implementation of the method as described in any one of claims 1-12, 13-20, or 21-25.

28. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions that, when executed by a communication device, implement the method as described in any one of claims 1-12, 13-20, or 21-25.

29. A computer program product, characterized in that, Includes computer program code, which, when run on a communication device, implements the method as described in any one of claims 1-12, 13-20, or 21-25.

30. A communication system, characterized in that, include: A data analysis network element for performing the method as described in any one of claims 1-12, a first apparatus for performing the method as described in any one of claims 13-20, and a first automatic dialing system for performing the method as described in any one of claims 21-25.

Images