Model and / or functionality evaluation method and apparatus, and device, product and storage medium

Abstract

Disclosed in the present disclosure are a model and / or functionality evaluation method and apparatus, and a related device, a product and a storage medium. The method is applied to a first device. The method comprises at least one of the following: performing model and / or functionality evaluation within a first time; and determining that a performance loss is less than or equal to a first threshold.

Description

Model and / or function evaluation method, device, apparatus, product and storage medium

[0001] Cross-reference to Related Applications

[0002] The present disclosure is based on and claims priority from Chinese Patent Application No. 202410866496.4, filed on June 28, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of wireless communication, and in particular to a model and / or function evaluation method, device, related apparatus, product and storage medium. BACKGROUND

[0004] In order to be able to complete the activation / deactivation and switching of an Artificial Intelligence (AI) or Machine Learning (ML) model in a timely manner, in addition to detecting and evaluating the AI / ML model currently in service, the performance of an inactive model also needs to be detected in real time or periodically to facilitate quick determination of a new model to be applied. However, there is a problem that some devices are limited in capability and may not be able to activate two or more models simultaneously. When evaluating the performance of an inactive model, the AI or ML model currently in application needs to be interrupted and switched to the inactive model for evaluation. Then, when the performance of the AI or ML model decreases, the network will not be able to determine whether the problem is caused by the mismatch of the current AI or ML model or by the switching of the terminal to the inactive model, and the network may make an incorrect decision, resulting in a decrease in system performance. SUMMARY

[0005] To solve the existing technical problems, the embodiments of the present disclosure provide a model and / or function evaluation method, device, related apparatus, product and storage medium.

[0006] The embodiments of the present disclosure provide a model and / or function evaluation method, which is performed by a first device, and the method includes at least one of the following:

[0007] Model and / or function evaluation is performed within a first time;

[0008] It is determined that the performance loss is less than or equal to a first threshold.

[0009] In some embodiments, the model includes one of the following:

[0010] An inactive model

[0011] an artificial intelligence, AI, and / or machine learning, ML, model;

[0012] an inactive AI and / or ML model; and / or,

[0013] the function comprises one of:

[0014] an inactive function;

[0015] an artificial intelligence, AI, and / or machine learning, ML, function;

[0016] an inactive AI and / or ML function.

[0017] In some embodiments, the first time comprises at least one of a time for a model of the service to be converted to an inactive model, a time for evaluation of the model, and a time for the inactive model to be converted to the model of the service; and / or,

[0018] the first time comprises at least one of a time for a function of the service to be converted to an inactive function, a time for evaluation of the function, and a time for the inactive function to be converted to the function of the service.

[0019] In some embodiments, the method further comprises:

[0020] receiving a first message; the first message indicating at least one of:

[0021] a period of the first time;

[0022] an offset of the first time;

[0023] a length of the first time.

[0024] In some embodiments, the first time is composed of at least one time unit, and a bit information is used to indicate whether a first time unit within the first time is used for evaluation of a model and / or a function; the first time unit is any time unit of the at least one time unit; the bit information comprises at least one bit; one bit corresponds to one time unit; the bit has a first value and / or a second value; the first value represents that the first time unit is used for evaluation of the model and / or the function; the second value represents that the first time unit is not used for evaluation of the model and / or the function.

[0025] In some embodiments, the second time is constituted by at least one of the first times, and a bit information is used to indicate whether the first time is used for the evaluation of the model and / or the functionality in the second time; the first time is any one of the at least one first time; the bit information includes at least one bit; one bit corresponds to one first time; the bit has a first value and / or a second value; the first value represents that the first time is used for the evaluation of the model and / or the functionality; and the second value represents that the first time is not used for the evaluation of the model and / or the functionality.

[0026] In some embodiments, the method further includes:

[0027] sending second information; the second information is used to indicate that the first device completes the evaluation of the model; and / or, the second information is used to indicate that the first device completes the evaluation of the functionality.

[0028] In some embodiments, the second information includes at least one of:

[0029] indication information of completing the evaluation;

[0030] a model index ID;

[0031] a functionality index ID.

[0032] In some embodiments, the method further includes:

[0033] receiving third information;

[0034] the third information includes at least one of:

[0035] indicating that the first device performs the evaluation of the model in a measurement interval;

[0036] indicating that the first device performs the evaluation of the functionality in a measurement interval;

[0037] indicating that the first device does not perform the evaluation of the model in a measurement interval;

[0038] indicating that the first device does not perform the evaluation of the functionality in a measurement interval.

[0039] In some embodiments, the determination that the performance loss is less than or equal to the first threshold includes one of:

[0040] a loss rate of an acknowledgement character ACK and / or a negative acknowledgement character NACK is less than or equal to the first threshold;

[0041] The loss rate of the acknowledgement character ACK and / or the negative acknowledgement character NACK in the third time is less than or equal to the first threshold.

[0042] In some embodiments, the method further comprises:

[0043] sending fourth information; the fourth information comprises fourth time information of model and / or functionality evaluation of the first device;

[0044] The fourth time information comprises at least one of:

[0045] The period of the model and / or functionality evaluation;

[0046] The offset of the model and / or functionality evaluation;

[0047] The duration of the model and / or functionality evaluation.

[0048] The embodiments of the present disclosure further provide a model and / or functionality evaluation method, executed by a second device, comprising:

[0049] sending a first message to a first device; the first message indicates at least one of:

[0050] The period of the first time;

[0051] The offset of the first time;

[0052] The length of the first time.

[0053] In some embodiments, the first time comprises at least one of the time of service model conversion to inactive model, the evaluation time of model, and the time of inactive model conversion to service model; and / or,

[0054] The first time comprises at least one of the time of service functionality conversion to inactive functionality, the evaluation time of functionality, and the time of inactive functionality conversion to service functionality.

[0055] In some embodiments, the first time is composed of at least one time unit, and the evaluation of the model and / or the functionality in the first time unit is indicated by bit information; the first time unit is any time unit of the at least one time unit; the bit information includes at least one bit; one bit corresponds to one time unit; the value of the bit includes a first value and / or a second value; the first value represents that the first time unit is used for the evaluation of the model and / or the functionality; and the second value represents that the first time unit is not used for the evaluation of the model and / or the functionality.

[0056] In some embodiments, the second time is composed of at least one first time, and the evaluation of the model and / or the functionality in the first time is indicated by bit information; the first time is any first time of the at least one first time; the bit information includes at least one bit; one bit corresponds to one first time; the value of the bit includes a first value and / or a second value; the first value represents that the first time is used for the evaluation of the model and / or the functionality; and the second value represents that the first time is not used for the evaluation of the model and / or the functionality.

[0057] In some embodiments, the method further includes:

[0058] receiving second information sent by the first device; the second information is used to indicate that the first device completes the evaluation of the model; and / or, the second information is used to indicate that the first device completes the evaluation of the functionality.

[0059] In some embodiments, the second information includes at least one of:

[0060] indication information of completing the evaluation;

[0061] a model identification number ID;

[0062] a functionality identification number ID.

[0063] In some embodiments, the method further includes:

[0064] sending third information to the first device;

[0065] the third information includes at least one of:

[0066] indicating the first device to perform the evaluation of the model in a measurement interval;

[0067] indicating the first device to perform the evaluation of the functionality in a measurement interval;

[0068] indicating that the first device does not perform evaluation of the model in a measurement interval;

[0069] indicating that the first device does not perform evaluation of the functionality in a measurement interval.

[0070] In some embodiments, the method further comprises:

[0071] receiving fourth information sent by the first device; the fourth information comprising fourth time information of evaluation of the model and / or the functionality by the first device;

[0072] The fourth time information comprises at least one of:

[0073] a period of the evaluation of the model and / or the functionality;

[0074] an offset of the evaluation of the model and / or the functionality;

[0075] a length of the evaluation of the model and / or the functionality.

[0076] Embodiments of the present disclosure further provide a model and / or functionality evaluation apparatus arranged on a first device, comprising:

[0077] The apparatus comprises at least one of:

[0078] an evaluation unit configured to perform evaluation of a model and / or a functionality in a first time;

[0079] a performance unit configured to determine that a performance loss is less than or equal to a first threshold.

[0080] Embodiments of the present disclosure further provide a model and / or functionality evaluation apparatus arranged on a second device, comprising:

[0081] a sending unit configured to send a first message to the first device; the first message indicating at least one of:

[0082] a period of the first time;

[0083] an offset of the first time;

[0084] a length of the first time.

[0085] Embodiments of the present disclosure further provide a model and / or functionality evaluation apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any method on the first device side or the steps of any method on the second device side when executing the program.

[0086] The embodiments of the present disclosure further provide a computer program product, comprising a computer program which, when executed by a processor, implements the steps of any of the above-mentioned methods on the first device side, or implements the steps of any of the above-mentioned methods on the second device side.

[0087] The embodiments of the present disclosure further provide a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned methods on the first device side, or implements the steps of any of the above-mentioned methods on the second device side.

[0088] The model and / or function evaluation method, device, related equipment, product and storage medium provided by the embodiments of the present disclosure are applied to a first device, and the method comprises at least one of the following: performing model and / or function evaluation within a first time; and determining that a performance loss is less than or equal to a first threshold. By using the embodiments of the present disclosure, the model and / or function evaluation is performed within the first time; and / or, it is determined that the performance loss is less than or equal to the first threshold. That is, the first device performs model and / or function evaluation within the first time, and when AI / ML performance decreases within the first time, the network can know that the AI / ML performance decreases within the time period because of the AI / ML model evaluation, and then the network can accept the AI loss within the time period or does not require the terminal according to the AI performance requirement. If the AI / ML performance decreases outside the first time, the network can consider that the current AI / ML model is not suitable, and can trigger model switching, activation, deactivation, or even consider that the current environment is not suitable for AI, and instruct the terminal to fall back to a non-AI mode. At the same time, the throughput loss caused by the inactive AI / ML model evaluation is also considered, and the data loss rate allowed within the specified time is specified to constrain the freedom and throughput loss. BRIEF DESCRIPTION OF DRAWINGS

[0089] FIG. 1 is a schematic diagram of a model and / or function evaluation method according to an embodiment of the present disclosure;

[0090] FIG. 2 is a schematic diagram of another model and / or function evaluation method according to an embodiment of the present disclosure;

[0091] FIG. 3 is a schematic diagram of a model and / or function evaluation device according to an embodiment of the present disclosure;

[0092] FIG. 4 is a schematic diagram of another model and / or function evaluation device according to an embodiment of the present disclosure;

[0093] FIG. 5 is a schematic diagram of a first device according to an embodiment of the present disclosure;

[0094] FIG. 6 is a schematic diagram of a second device according to an embodiment of the present disclosure;

[0095] FIG. 7 is a schematic diagram of a model and / or function evaluation system structure according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0096] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments.

[0097] An AI or ML model can also be written as an AI / ML model. Due to changes in configured parameters or scenarios, the model no longer matches the environment, and the performance of the AI / ML model deteriorates. A special mechanism is needed to detect the performance of the model, and then help manage the life cycle of the model, including model activation / deactivation (or described as model switching).

[0098] AI / ML model activation / deactivation and switching need to evaluate whether the currently applied model is performance-degraded, and determine the target model to be activated / switched. For example, the currently working model is model A, and other non-applied models are model B, model C, and model D. When model A is no longer applicable, it is necessary to determine which one of models B, C, and D is applicable to the current environment. The non-applied models B, C, and D belong to inactive models. In order to complete the activation / deactivation / switching of the AI / ML model in a timely manner, in addition to detecting and evaluating the currently served AI / ML model, it is necessary to detect the performance of the inactive model in real time or periodically, so as to quickly determine the new applied model.

[0099] However, there is a problem that, depending on the device capability, when the performance of the inactive model is evaluated, the currently applied AI / ML model needs to be interrupted and switched to the inactive model for evaluation, which will cause interruption of the AI service during the switching and evaluation processes. Moreover, if the terminal decides at any time when to evaluate the inactive model, when the AI / ML performance deteriorates, the network cannot determine whether the problem is caused by the mismatch of the current AI / ML model or the switching of the terminal to the inactive model, which may cause the network to make a wrong decision. In summary, the terminal behavior is uncontrollable, the network and the terminal do not understand the AI / ML performance deterioration, and the system performance deteriorates.

[0100] An embodiment of the present disclosure provides a model and / or function evaluation method, which is applied to a first device. As shown in FIG. 1, FIG. 1 is a schematic diagram of a method for evaluating a model and / or function according to an embodiment of the present disclosure. The method includes at least one of the following:

[0101] Step 101: performing model and / or function evaluation at a first time;

[0102] Step 102: Determine if the performance loss is less than or equal to the first threshold.

[0103] It should be noted that the first device can be determined according to the actual situation, and is not limited here. As an example, the first device may include a terminal and / or network-side equipment, such as a base station or a location management function (LMF).

[0104] In step 101, the first time can be determined according to the actual situation, and is not limited here. The first time can also be described as a first duration, which can be understood as a range of time.

[0105] The model can be determined based on the actual situation and is not limited here. As an example, the model can be an artificial intelligence (AI) and / or machine learning (ML) model. It can be simply referred to as model, AI, and / or ML model.

[0106] The functions described can be determined based on actual conditions and are not limited here. As an example, the functions can be described as functionalities, or AI and / or ML functionalities. A functionality may include or correspond to at least one model. A functionality applies to or corresponds to a certain scenario, such as beam management (or described as beam prediction, including temporal prediction and spatial prediction), CSI compression, CSI prediction, and positioning (including direct positioning and assisted positioning). A model applies to or corresponds to a certain configuration, such as different antenna configurations (e.g., the number of antenna ports), different beam counts, and different stream counts will correspond to different models. Evaluation can also be described as monitoring, performance monitoring, or lifecycle management (LCM). Model evaluation includes the evaluation of deactivated models, and functional evaluation includes the evaluation of deactivated models. Deactivation can also be described as not being deployed or not being applied.

[0107] In step 102, the performance loss can be determined based on the actual situation and is not limited here. As an example, the performance loss can be described as lost ACK / NACK, or as throughput loss. The performance loss indicates the performance loss due to model and / or functionality evaluation. Determining that the performance loss is less than or equal to a first threshold can also be described as the performance loss not exceeding a first threshold within a certain period of time. The first threshold can be a proportion, percentage, or probability.

[0108] The advantage of the scheme that performs model and / or functional evaluation in the first instance is that it specifies when and where the first device performs the evaluation. The first and second devices can have a consistent understanding. Taking the network and the terminal as an example, they can agree on when to perform inactive AI / ML model evaluation. Evaluation is performed at the time specified by the network, transitioning from a serving AI / ML model to an inactive AI / ML model. Even if AI performance degrades, the network can determine that it's due to the model transition, and the network can make corresponding adjustments within that timeframe. If AI / ML performance degrades within the network-specified timeframe, the network knows that the degradation is due to AI / ML model evaluation during that period, and can accept the AI ​​loss during that time, or not require the terminal to meet AI performance requirements (meaning the network doesn't need to take additional measures). If AI / ML performance degrades outside the network-specified timeframe, the network may consider the current AI / ML model unsuitable, potentially triggering model switching, activation, or deactivation prematurely. Furthermore, the network may consider the current environment unsuitable for AI and instruct the terminal to revert to a non-AI mode.

[0109] The gain of the scheme where the performance loss is less than or equal to the first threshold is as follows: Compared to the first-time scheme, this scheme does not specify the exact location for evaluation, giving the terminal more freedom to determine how to perform inactive AI / ML model evaluation. Simultaneously, considering the throughput loss caused by inactive AI / ML model evaluation, a constraint is imposed by specifying the allowable data loss rate within a predetermined timeframe, achieving a trade-off between freedom and throughput loss. The first-time scheme and the first-threshold scheme can be used independently or in combination.

[0110] In this embodiment, model and / or functional evaluation is performed within a first time period; and / or, the performance loss is determined to be less than or equal to a first threshold. That is, the first device performs model and / or functional evaluation at the first time period. If AI / ML performance degradation occurs within this first time period, the network knows that the degradation is due to AI / ML model evaluation during that period. Therefore, the network can accept the AI ​​performance loss during that period, or may not require the terminal to meet AI performance requirements. If AI / ML performance degradation occurs outside the first time period, the network may consider the current AI / ML model unsuitable, potentially triggering model switching, activation, or deactivation prematurely. Furthermore, the network may consider the current environment unsuitable for AI and instruct the terminal to revert to a non-AI mode. Simultaneously, considering the throughput loss caused by inactive AI / ML model evaluation, a constraint is imposed by specifying an allowable data loss rate within a predetermined time period, achieving a trade-off between degrees of freedom and throughput loss.

[0111] In one embodiment, the model includes one of the following:

[0112] Deactivate the model (inactive model);

[0113] Artificial intelligence (AI) and / or machine learning (ML) models;

[0114] Deactivated AI and / or ML models; and / or,

[0115] The function includes one of the following:

[0116] Deactivate functionality;

[0117] Artificial intelligence (AI) and / or machine learning (ML) capabilities;

[0118] Deactivate AI and / or ML functionality.

[0119] In this embodiment, the artificial intelligence (AI) and / or machine learning (ML) model can also be referred to as an AI and / or ML model, including AI model, ML model, AI model and ML model.

[0120] The deactivated AI and / or ML model may include a deactivated AI model, a deactivated ML model, and a deactivated AI model and ML model.

[0121] The artificial intelligence (AI) and / or machine learning (ML) functions may include AI functions, ML functions, and de-AI and / or ML functions.

[0122] The deactivated AI and / or ML functionality may include deactivated AI functionality, deactivated ML functionality, and deactivated AI and ML functionality.

[0123] In one embodiment, the first time includes at least one of the following: the time for the service's model to transition to an inactive model, the model's evaluation time, and the time for the inactive model to transition back to the service's model; and / or,

[0124] The first time includes at least one of the following: the time when the service's functionality is converted to inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality is converted to the service's functionality.

[0125] In this embodiment, the first time includes at least one of the following: the time when the service model is converted to the inactive model, the evaluation time of the model, and the time when the inactive model is converted to the service model. This can be understood as the first time including the time when the service model is converted to the inactive model, the evaluation time of the model, the time when the inactive model is converted to the service model, the time when the service model is converted to the inactive model and the evaluation time of the model, the time when the service model is converted to the inactive model and the evaluation time of the model, and the time when the inactive model is converted to the service model.

[0126] The first time includes at least one of the following: the time when the service's functionality is converted to inactive functionality, the evaluation time of the functionality, and the time when inactive functionality is converted to the service's functionality. This can be understood as the first time including the time when the service's functionality is converted to inactive functionality, the evaluation time of the functionality, the time when inactive functionality is converted to the service's functionality, the time when the service's functionality is converted to inactive functionality and the evaluation time of the functionality, the time when the service's functionality is converted to inactive functionality and the evaluation time of the functionality, and the time when inactive functionality is converted to the service's functionality.

[0127] In practical applications, the service can also be described as being applied, deployed, or currently applied / deployed.

[0128] In one embodiment, the method further includes:

[0129] Receive a first message; the first message indicates at least one of the following:

[0130] The period of the first time;

[0131] The offset of the first time;

[0132] The length of the first time period.

[0133] In this embodiment, the first message can be determined according to the actual situation and is not limited here. As an example, the first message can be sent by a network-side device, a core network, an LMF, or an OTT (Over-The-Top) server. If the first device is a terminal, the first message can be sent by at least one of the network-side device, the core network, the LMF, or the OTT server; if the first device is a network-side device, the first message can be sent by at least one of the core network, the LMF, or the OTT server.

[0134] The period of the first time can be determined according to the actual situation and is not limited here. As an example, the period of the first time can also be described as the period of model and / or functional evaluation.

[0135] The offset of the first time can be determined according to the actual situation and is not limited here. As an example, the offset of the first time can also be described as the starting position of the model and / or functional evaluation time, that is, where in the cycle.

[0136] The length of the first time interval can be determined based on actual circumstances and is not limited here. As an example, the length of the first time interval can also be described as the duration of the first time interval or as the duration of the first time interval. The length of the first time interval includes at least the time during which the first device performs model evaluation.

[0137] Receiving the first message can be understood as the first device receiving the first message sent by the second device. The second device can be determined based on actual circumstances and is not limited here. As an example, the second device can be a network-side device, such as a base station or network.

[0138] In practical applications, the terminal receives the first information sent by the network. The first information includes the time information of the terminal's inactive model evaluation. Specifically, the time information can be a period of time within a certain duration, during which the terminal only evaluates the inactive AI / ML model (it can also include the conversion time from the currently served AI / ML model to the inactive AI / ML model, and the conversion time from the inactive AI / ML model to the original service AI / ML model).

[0139] Furthermore, the time information included in the first information can be periodic. In this case, the first information includes the period, offset, and duration (or can be described as the evaluation duration) of the inactive AI / ML model evaluation.

[0140] In one embodiment, the first time period is constituted by at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and / or functional evaluation; the first time unit is any one of the at least one time units; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and / or a second value; the first value indicates that the first time unit is used for model and / or functional evaluation; the second value indicates that the first time unit is not used for model and / or functional evaluation.

[0141] In this embodiment, the step of indicating whether or not the first time unit performs model and / or function evaluation within the first time period through bit information can be determined according to the actual situation and is not limited here. As an example, indicating whether or not the first time unit performs model and / or function evaluation through bit information can be understood as indicating whether or not the first time unit is used to perform model and / or function evaluation within the first time period. Here, "yes" or "no" can be described using "whether".

[0142] In practical applications, the time information included in the first information can also be non-periodic. In this case, the first information includes one or more location information for inactive AI / ML model evaluation within a certain time period. Specifically, as one implementation, a certain duration can be composed of multiple time units. A bit string indicates which locations (time units) within this duration can perform AI / ML inactive model evaluation. One bit corresponds to one time unit. A bit value of 1 or TRUE indicates that the time unit can be used for inactive model evaluation (or can be described as AI / ML model that can interrupt the current service). A bit value of 0 or FAULSE indicates that the time unit cannot be used for inactive model evaluation (or can be described as AI / ML model that cannot interrupt the current service).

[0143] In one embodiment, a second time is constituted by at least one first time, and bit information is used to indicate whether or not a model and / or function evaluation is performed during the second time; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and / or a second value; the first value indicates that the first time is used for model and / or function evaluation; the second value indicates that the first time is not used for model and / or function evaluation.

[0144] In this embodiment, the specific process of constructing the second time from at least one first time can be determined according to the actual situation and is not limited here. As an example, the second time can be understood as a larger time range; that is, the second time can be understood as being composed of N first times, or as indicating the position of the first time that can be evaluated within a length equal to the length of N first times (N is an integer). The first time can be understood as being within a smaller time range; that is, a subset of the first time, which can be understood as being composed of M first time units, or as indicating the position of the first time unit that can be evaluated within a length equal to the length of M first time units.

[0145] In practical applications, a scheme can be defined as follows: The first time period consists of at least one time unit, and bit information is used to indicate whether or not a model and / or functional evaluation is performed within that first time unit. Alternatively, a second time period can be defined by at least one first time unit, and bit information is used to indicate whether or not a model and / or functional evaluation is performed within that second time unit. These two schemes can be used individually or in combination. Both schemes enable aperiodic evaluation. The bit information can be sent to the first device via a message.

[0146] In one embodiment, the method further includes:

[0147] If the performance of the inactive AI and / or ML model degrades within a preset time period, information on the performance loss of the inactive AI and / or ML model within the preset time period is sent to the second device.

[0148] In this embodiment, the preset duration and the loss information can be determined according to the actual situation, and are not limited here.

[0149] In one embodiment, the method further includes:

[0150] If the performance of the inactive AI and / or ML model degrades outside the preset time period, the system receives an indication message sent by the terminal.

[0151] Based on the indicated information, the system reverts to a mode that does not correspond to the inactive AI and / or ML model.

[0152] In this embodiment, the indication information can be determined according to the actual situation, and is not limited here. As an example, the indication information is used to fall back to a mode that does not correspond to the inactive AI and / or ML model.

[0153] In one embodiment, the method further includes:

[0154] Send a second message; the second message is used to instruct the first device to complete the evaluation of the model; and / or, the second message is used to instruct the first device to complete the evaluation of the function.

[0155] In this embodiment, the second information can be determined according to the actual situation, and is not limited here. As an example, the second information includes at least one of the following: indication information for completing the evaluation; model index ID; function index ID.

[0156] In practical applications, the terminal receives a second message from the network, indicating whether or not the terminal can perform inactive AI / ML model evaluation within the measurement interval. If the second message indicates that the terminal can perform inactive AI / ML model evaluation within the measurement interval, then the aforementioned first message does not need to be sent; if the second message indicates that the terminal cannot perform inactive AI / ML model evaluation within the measurement interval, then the aforementioned first message needs to be sent again to indicate where the evaluation should be performed.

[0157] Specifically, when mobility requirements are low, such as when there are few frequency points to be measured that require measurement intervals, the network can be configured to use measurement intervals for inactive AI / ML model evaluation, that is, to exchange the extended measurement time for a small throughput loss caused by AI / ML model evaluation.

[0158] As an example, the second information may include a gain of the second information, mainly considering that the evaluation time required for different AI / ML models varies. To cover as many scenarios as possible, taking network and terminal scenarios as an example, the network usually configures a longer duration for evaluation. However, for models that do not require a long evaluation time, this longer allowed evaluation time is a waste. The network cannot know when the terminal has completed the evaluation of a model. By introducing the second information, when the AI / ML model evaluation is completed, the terminal is allowed to notify the network. Then the network knows that the terminal has resumed AI service for the remaining time of the configured duration.

[0159] In practical applications, the benefit of the second information is that the network does not need to be configured separately for the time information used for evaluating inactive AI / ML models, which can reduce signaling overhead and reduce throughput loss.

[0160] In one embodiment, the second information includes at least one of the following:

[0161] Instructions for completing the assessment;

[0162] Model Index ID;

[0163] Function Index ID.

[0164] In this embodiment, the indication information can be determined according to the actual situation and is not limited here. As an example, the indication information indicates that an inactive AI / ML model evaluation should be performed within the measurement interval.

[0165] Both the model index ID and the function index ID can be determined based on actual circumstances, and are not limited here. The model index ID identifies the specific model; the function index ID identifies the specific function.

[0166] In one embodiment, the method further includes:

[0167] Receive third-party information;

[0168] The third information includes at least one of the following:

[0169] Instruct the first device to evaluate the model within the measurement interval;

[0170] Instruct the first device to perform a functionality evaluation within the measurement interval;

[0171] Instruct the first device not to evaluate the model during the measurement interval;

[0172] This instructs the first device not to perform a functionality evaluation during the measurement interval.

[0173] In this embodiment, the third information includes at least one of the following: instructing the first device to perform model evaluation within the measurement interval; instructing the first device to perform functionality evaluation within the measurement interval; instructing the first device not to perform model evaluation within the measurement interval; instructing the first device not to perform functionality evaluation within the measurement interval; which can be understood as instructing whether the first device performs model or functionality evaluation within the measurement interval. That is, the third information can instruct whether the first device performs model or functionality evaluation within the measurement interval.

[0174] The third information indicating whether or not the first device performs model or functionality evaluation within the measurement interval can also be referred to as the third information indicating whether / not the first device performs model or functionality evaluation within the measurement interval. "Whether or not the first device performs model / functionality evaluation within the measurement interval" can also be described as whether / not the measurement interval can be used for model / functionality evaluation. Here, the measurement interval is a measurement interval used for other purposes, such as the measurement of existing measurement targets. When mobility requirements are low, and if the required measurement interval has few frequency points (frequency points can also be described as measurement targets), the network can configure the terminal to use this measurement interval for model / functionality evaluation, i.e., trading off a smaller throughput loss due to model / functionality evaluation by extending the measurement time.

[0175] Receiving third information can be the process by which the first device receives third information sent by the second device.

[0176] In practical applications, the first device can be a terminal; the second device can be a network; the terminal receives third information sent by the network, which includes a configured period, i.e., compared to the first information, the third information indicates a duration, such as X milliseconds or Y seconds. The network does not specify the specific location where the terminal performs AI / ML model evaluation within this duration, which is determined autonomously by the terminal. However, the maximum system loss allowed within this time range needs to be pre-defined in the protocol. For example, the maximum allowed ACK / NACK loss rate within X milliseconds or Y seconds is Z%.

[0177] The third piece of information can be determined based on the actual situation and is not limited here. As an example, compared with the first piece of information, the third piece of information gives the terminal more freedom to determine how to perform inactive AI / ML model evaluation. At the same time, considering the throughput loss caused by inactive AI / ML model evaluation, a constraint is imposed by specifying the allowable data loss rate within a predetermined time period, thus achieving a trade-off between the degree of freedom and the throughput loss.

[0178] The benefit of the third information is that the network does not need to be configured separately for time information used for model / functionality evaluation, which can reduce signaling overhead and reduce throughput loss.

[0179] In one embodiment, determining that the performance loss is less than or equal to a first threshold includes one of the following:

[0180] The loss rate of the ACK and / or NACK characters is less than or equal to the first threshold;

[0181] The loss rate of ACK and / or NACK characters within the third time period is less than or equal to the first threshold.

[0182] In this embodiment, both the first threshold and the third time can be determined according to the actual situation, and no limitation is made here.

[0183] In one embodiment, the method further includes:

[0184] Send a fourth message; the fourth message includes fourth time information of the first device performing model and / or functional evaluation;

[0185] The fourth time information includes at least one of the following:

[0186] The cycle of the model and / or functional evaluation;

[0187] The offset of the model and / or functional evaluation;

[0188] The duration of the model and / or functional evaluation.

[0189] In this embodiment, the offset of the model and / or function evaluation can be determined according to the actual situation, and is not limited here. As an example, the offset of the model and / or function evaluation can be understood as the specific evaluation position of the model and / or function evaluation.

[0190] The duration of the model and / or function evaluation can be determined according to the actual situation and is not limited here. As an example, the duration of the model and / or function evaluation can be simply referred to as the evaluation duration.

[0191] Sending the fourth information can be understood as the terminal sending the fourth information to the network.

[0192] The fourth information includes the fourth time information of the first device performing model and / or functional evaluation, which can be understood as the fourth information including the time information of the terminal's expected inactive AI / ML model evaluation.

[0193] In practical applications, the terminal sends a fourth piece of information to the network. This fourth piece of information includes the terminal's desired time for evaluating inactive AI / ML models, such as the period, specific evaluation location, and evaluation duration. The network can refer to this fourth piece of information to configure the first, second, and third pieces of information.

[0194] The fourth time information can be determined according to the actual situation and is not limited here. As an example, the fourth time information may include the gain of the fourth information: Since the second device cannot know the situation of the first device, the configuration is not optimal. The first device can determine how to evaluate according to its own environment and report the information to the second device through the fourth information to assist the second device in configuring the first time, bit information and the first threshold.

[0195] In practical applications, the benefit of the fourth information lies in the fact that since the network cannot know the terminal status, the configuration is not optimal. Finally, it can determine how to evaluate based on its own environment and report this information to the network through the fourth information to assist the network in configuration.

[0196] In one embodiment, the method further includes:

[0197] Obtain the environmental information of the terminal;

[0198] Based on the environmental information and the second time information, an inactive AI and / or ML model is evaluated to obtain the evaluation results;

[0199] The evaluation result is sent to the network device; the evaluation result is used by the network device to update the configuration message.

[0200] In this embodiment of the disclosure, the environmental information of the terminal is obtained; wherein, the environmental information can be understood as the environmental conditions in which the terminal is located.

[0201] The embodiments disclosed herein mainly consider that the network configuration may not be optimal due to the inability to know the terminal's status. The terminal can determine how to evaluate based on its own environment and report this information to the network to assist the network in configuration.

[0202] Accordingly, this disclosure also provides a model and / or function evaluation method, as shown in FIG2. FIG2 is a schematic flowchart of another model and / or function evaluation method according to an embodiment of this disclosure, applied to a second device. The method includes:

[0203] Step 201: Send a first message to the first device; the first message indicates at least one of the following:

[0204] The first time period;

[0205] The offset of the first time;

[0206] The length of the first time period.

[0207] It should be noted that the first device can be determined according to the actual situation, and is not limited here. As an example, the first device may include a terminal and / or network-side equipment, such as a base station or LMF.

[0208] The second device can be determined according to the actual situation, and is not limited here. As an example, the second device can be a network-side device, such as a base station or network.

[0209] In step 201, the first message can be determined according to the actual situation, and is not limited here. As an example, the first message can be sent by a network-side device, a core network, an LMF, or an OTT server. If the first device is a terminal, then the first message can be sent by at least one of the network-side device, the core network, the LMF, and the OTT server; if the first device is a network-side device, then the first message can be sent by at least one of the core network, the LMF, and the OTT server.

[0210] The period of the first time can be determined according to the actual situation and is not limited here. As an example, the period of the first time can also be described as the period of model and / or functional evaluation.

[0211] The offset of the first time can be determined according to the actual situation and is not limited here. As an example, the offset of the first time can also be described as the starting position of the model and / or functional evaluation time, that is, where in the cycle.

[0212] The length of the first time interval can be determined based on actual circumstances and is not limited here. As an example, the length of the first time interval can also be described as the duration of the first time interval or as the duration of the first time interval. The length of the first time interval includes at least the time during which the first device performs model evaluation.

[0213] Receiving the first message can be understood as the first device receiving the first message sent by the second device. The second device can be determined based on actual circumstances and is not limited here. As an example, the second device can be a network-side device, such as a base station or network.

[0214] In practical applications, the terminal receives the first information sent by the network. The first information includes the time information of the terminal's inactive model evaluation. Specifically, the time information can be a period of time within a certain duration, during which the terminal only evaluates the inactive AI / ML model (it can also include the conversion time from the currently served AI / ML model to the inactive AI / ML model, and the conversion time from the inactive AI / ML model to the original service AI / ML model).

[0215] Furthermore, the time information included in the first information can be periodic. In this case, the first information includes the period, offset, and duration (or can be described as the evaluation duration) of the inactive AI / ML model evaluation.

[0216] In one embodiment, the first time includes at least one of the following: the time for the service's model to transition to an inactive model, the model's evaluation time, and the time for the inactive model to transition back to the service's model; and / or,

[0217] The first time includes at least one of the following: the time when the service's functionality is converted to inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality is converted to the service's functionality.

[0218] In this embodiment, the first time includes at least one of the following: the time when the service model is converted to the inactive model, the evaluation time of the model, and the time when the inactive model is converted to the service model. This can be understood as the first time including the time when the service model is converted to the inactive model, the evaluation time of the model, the time when the inactive model is converted to the service model, the time when the service model is converted to the inactive model and the evaluation time of the model, the time when the service model is converted to the inactive model and the evaluation time of the model, and the time when the inactive model is converted to the service model.

[0219] The first time includes at least one of the following: the time when the service's functionality is converted to inactive functionality, the evaluation time of the functionality, and the time when inactive functionality is converted to the service's functionality. This can be understood as the first time including the time when the service's functionality is converted to inactive functionality, the evaluation time of the functionality, the time when inactive functionality is converted to the service's functionality, the time when the service's functionality is converted to inactive functionality and the evaluation time of the functionality, the time when the service's functionality is converted to inactive functionality and the evaluation time of the functionality, and the time when inactive functionality is converted to the service's functionality.

[0220] In practical applications, the service can also be described as being applied, deployed, or currently applied / deployed.

[0221] In one embodiment, the first time period is constituted by at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and / or functional evaluation; the first time unit is any one of the at least one time units; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and / or a second value; the first value indicates that the first time unit is used for model and / or functional evaluation; the second value indicates that the first time unit is not used for model and / or functional evaluation.

[0222] In this embodiment, the step of indicating whether or not the first time unit performs model and / or function evaluation within the first time period through bit information can be determined according to the actual situation and is not limited here. As an example, indicating whether or not the first time unit performs model and / or function evaluation through bit information can be understood as indicating whether or not the first time unit is used to perform model and / or function evaluation within the first time period. Here, "yes" or "no" can be described using "whether".

[0223] In practical applications, the time information included in the first information can also be non-periodic. In this case, the first information includes one or more location information for inactive AI / ML model evaluation within a certain time period. Specifically, as one implementation, a certain duration can be composed of multiple time units. A bit string indicates which locations (time units) within this duration can perform AI / ML inactive model evaluation. One bit corresponds to one time unit. A bit value of 1 or TRUE indicates that the time unit can be used for inactive model evaluation (or can be described as AI / ML model that can interrupt the current service). A bit value of 0 or FAULSE indicates that the time unit cannot be used for inactive model evaluation (or can be described as AI / ML model that cannot interrupt the current service).

[0224] In one embodiment, a second time is constituted by at least one first time, and bit information is used to indicate whether or not a model and / or function evaluation is performed during the second time; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and / or a second value; the first value indicates that the first time is used for model and / or function evaluation; the second value indicates that the first time is not used for model and / or function evaluation.

[0225] In this embodiment, the specific process of constructing the second time from at least one first time can be determined according to the actual situation and is not limited here. As an example, the second time can be understood as a larger time range; that is, the second time can be understood as being composed of N first times, or as indicating the position of the first time that can be evaluated within a length equal to the length of N first times (N is an integer). The first time can be understood as being within a smaller time range; that is, a subset of the first time, which can be understood as being composed of M first time units, or as indicating the position of the first time unit that can be evaluated within a length equal to the length of M first time units.

[0226] In practical applications, a scheme can be defined as follows: The first time period consists of at least one time unit, and bit information is used to indicate whether or not a model and / or functional evaluation is performed within that first time unit. Alternatively, a second time period can be defined by at least one first time unit, and bit information is used to indicate whether or not a model and / or functional evaluation is performed within that second time unit. These two schemes can be used individually or in combination. Both schemes enable aperiodic evaluation. The bit information can be sent to the first device via a message.

[0227] In one embodiment, the method further includes:

[0228] The system receives second information sent by the first device; the second information is used to instruct the first device to complete the evaluation of the model; and / or, the second information is used to instruct the first device to complete the evaluation of the function.

[0229] In this embodiment, the second information can be determined according to the actual situation, and is not limited here. As an example, the second information includes at least one of the following: indication information for completing the evaluation; model index ID; function index ID.

[0230] Receiving the second information sent by the first device can be understood as the second device receiving the second information sent by the first device.

[0231] In practical applications, the first device can be a terminal; the second device can be a network; the network receives second information sent by the terminal, the second information indicating whether the terminal can perform inactive AI / ML model evaluation within the measurement interval. If the second information indicates that the terminal can perform inactive AI / ML model evaluation within the measurement interval, then the aforementioned first information need not be sent; if the second information indicates that the terminal cannot perform inactive AI / ML model evaluation within the measurement interval, then the aforementioned first information needs to be sent to indicate where the terminal should perform the evaluation.

[0232] Specifically, when mobility requirements are low, such as when there are few frequency points to be measured that require measurement intervals, the network can be configured to use measurement intervals for inactive AI / ML model evaluation, that is, to exchange the extended measurement time for a small throughput loss caused by AI / ML model evaluation.

[0233] As an example, the second information may include a gain of the second information, mainly considering that the evaluation time required for different AI / ML models varies. To cover as many scenarios as possible, taking network and terminal scenarios as an example, the network usually configures a longer duration for evaluation. However, for models that do not require a long evaluation time, this longer allowed evaluation time is a waste. The network cannot know when the terminal has completed the evaluation of a model. By introducing the second information, when the AI / ML model evaluation is completed, the terminal is allowed to notify the network. Then the network knows that the terminal has resumed AI service for the remaining time of the configured duration.

[0234] In practical applications, the benefit of the second information is that the network does not need to be configured separately for the time information used for evaluating inactive AI / ML models, which can reduce signaling overhead and reduce throughput loss.

[0235] In one embodiment, the second information includes at least one of the following:

[0236] Instructions for completing the assessment;

[0237] Model Identifier ID;

[0238] Functional Identifier ID.

[0239] In this embodiment, the indication information can be determined according to the actual situation and is not limited here. As an example, the indication information indicates that an inactive AI / ML model evaluation should be performed within the measurement interval.

[0240] Both the model index ID and the function index ID can be determined based on actual circumstances, and are not limited here. The model index ID identifies the specific model; the function index ID identifies the specific function.

[0241] In one embodiment, the method further includes:

[0242] Send the third information to the first device;

[0243] The third information includes at least one of the following:

[0244] Instruct the first device to evaluate the model within the measurement interval;

[0245] Instruct the first device to perform a functionality evaluation within the measurement interval;

[0246] Instruct the first device not to evaluate the model during the measurement interval;

[0247] This instructs the first device not to perform a functionality evaluation during the measurement interval.

[0248] In this embodiment, the third information includes at least one of the following: instructing the first device to perform model evaluation within the measurement interval; instructing the first device to perform functionality evaluation within the measurement interval; instructing the first device not to perform model evaluation within the measurement interval; instructing the first device not to perform functionality evaluation within the measurement interval; which can be understood as instructing whether the first device performs model or functionality evaluation within the measurement interval. That is, the third information can instruct whether the first device performs model or functionality evaluation within the measurement interval.

[0249] The third information indicating whether or not the first device performs model or functionality evaluation within the measurement interval can also be referred to as the third information indicating whether / not the first device performs model or functionality evaluation within the measurement interval. The phrase "whether / not the first device performs model / functionality evaluation within the measurement interval" can also be described as whether / not the measurement interval can be used for model / functionality evaluation. Here, the measurement interval refers to a measurement interval used for other purposes, such as the measurement of a target in existing technologies. When mobility requirements are low, and if the required measurement interval has few frequency points (frequency points can also be described as measurement targets), the network can configure the terminal to use this measurement interval for model / functionality evaluation, i.e., trading off a smaller throughput loss due to model / functionality evaluation by extending the measurement time.

[0250] Sending third information to the first device can be a process where the second device sends third information to the first device.

[0251] In practical applications, the first device can be a terminal; the second device can be a network; the network sends third information to the terminal, which includes a configured period, i.e., compared to the first information, the third information indicates a duration, such as X milliseconds or Y seconds. The network does not specify the specific location where the terminal performs AI / ML model evaluation within this duration, which is determined autonomously by the terminal. However, the maximum allowable system loss within this time range needs to be pre-defined in the protocol. For example, the maximum allowable ACK / NACK loss rate within X milliseconds or Y seconds is Z%.

[0252] The third piece of information can be determined based on the actual situation and is not limited here. As an example, compared with the first piece of information, the third piece of information gives the terminal more freedom to determine how to perform inactive AI / ML model evaluation. At the same time, considering the throughput loss caused by inactive AI / ML model evaluation, a constraint is imposed by specifying the allowable data loss rate within a predetermined time period, thus achieving a trade-off between the degree of freedom and the throughput loss.

[0253] The benefit of the third information is that the network does not need to be configured separately for time information used for model / functionality evaluation, which can reduce signaling overhead and reduce throughput loss.

[0254] In one embodiment, the method further includes:

[0255] Receive fourth information sent by the first device; the fourth information includes fourth time information of the first device performing model and / or functional evaluation;

[0256] The fourth time information includes at least one of the following:

[0257] The cycle of the model and / or functional evaluation;

[0258] The offset of the model and / or functional evaluation;

[0259] The duration of the model and / or functional evaluation.

[0260] In this embodiment, the offset of the model and / or function evaluation can be determined according to the actual situation, and is not limited here. As an example, the offset of the model and / or function evaluation can be understood as the specific evaluation position of the model and / or function evaluation.

[0261] The duration of the model and / or function evaluation can be determined according to the actual situation and is not limited here. As an example, the duration of the model and / or function evaluation can be simply referred to as the evaluation duration.

[0262] Receiving the fourth information sent by the first device can be understood as the second device receiving the fourth information sent by the first device.

[0263] The fourth information includes the fourth time information of the first device performing model and / or functional evaluation, which can be understood as the fourth information including the time information of the terminal's expected inactive AI / ML model evaluation.

[0264] In practical applications, the first device can be a terminal; the first device can be a network; the network receives the fourth information sent by the terminal, which includes the terminal's desired time information for evaluating inactive AI / ML models, including the period, specific evaluation location, and evaluation duration. The network can refer to this fourth information to configure the first, second, and third information.

[0265] The fourth time information can be determined according to the actual situation and is not limited here. As an example, the fourth time information may include the gain of the fourth information: Since the second device cannot know the situation of the first device, the configuration is not optimal. The first device can determine how to evaluate according to its own environment and report the information to the second device through the fourth information to assist the second device in configuring the first time, bit information and the first threshold.

[0266] In practical applications, the benefit of the fourth information lies in the fact that since the network cannot know the terminal status, the configuration is not optimal. Finally, it can determine how to evaluate based on its own environment and report this information to the network through the fourth information to assist the network in configuration.

[0267] For ease of understanding, this disclosure provides a specific scheme for example models and / or functional evaluation methods.

[0268] Option 1: The terminal receives the first information sent by the network. The first information includes the time information for the terminal to evaluate the inactive model. Specifically, the time information can be a period of time within a certain duration, during which the terminal only evaluates the inactive AI / ML model (it can also include the conversion time from the currently served AI / ML model to the inactive AI / ML model, and the conversion time from the inactive AI / ML model to the original service AI / ML model).

[0269] Furthermore, the time information included in the first information can be periodic. In this case, the first information includes the period, offset, and duration (or can be described as the evaluation duration) of the inactive AI / ML model evaluation.

[0270] The time information included in the first information can also be aperiodic. In this case, the first information includes one or more location information for inactive AI / ML model evaluation within a certain time period. Specifically, as one implementation, a certain duration can be composed of multiple time units. A bit string indicates which locations (time units) within this duration can perform AI / ML inactive model evaluation. One bit corresponds to one time unit. A bit value of 1 or TRUE indicates that the time unit can be used for inactive model evaluation (or can be described as AI / ML model that can interrupt the current service), while a bit value of 0 or FAULSE indicates that the time unit cannot be used for inactive model evaluation (or can be described as AI / ML model that cannot interrupt the current service).

[0271] Furthermore, the terminal sends a fifth message to the network, indicating that the model evaluation is complete. Optionally, the fifth message may also include the model ID or functionality identifier that the evaluation is complete.

[0272] The benefit of introducing the fifth piece of information lies in the fact that, if the first piece of information includes the duration of evaluation, considering the different evaluation times required for different AI / ML models, and to cover as many scenarios as possible, the network usually configures a longer duration for evaluation. However, for models that do not require a long evaluation time, this longer allowed evaluation time is a waste. Furthermore, the network cannot know when the terminal has completed the evaluation of a model. By introducing the fifth piece of information, when the AI / ML model evaluation is completed, the terminal is allowed to notify the network. Then the network knows that the terminal has resumed AI services for the remaining time of the configured duration.

[0273] The first information gain is that the network and the terminal have a consistent understanding of when to perform inactive AI / ML model evaluation. The evaluation is carried out at the time position configured by the network to switch from service AI / ML model to inactive AI / ML model. Even if the AI ​​performance degrades, the network can determine that it is due to the model switch, and the network can make corresponding adjustments during this period.

[0274] Network-side behavior: If AI / ML performance degradation occurs within the network-configured timeframe, the network recognizes that this degradation is due to AI / ML model evaluation. Therefore, the network can accept the AI ​​performance loss during this period or may not require the terminal to meet AI performance demands (meaning the network doesn't need to take additional measures). If AI / ML performance degradation occurs outside the network-configured timeframe, the network may deem the current AI / ML model unsuitable and prematurely trigger model switching, activation, or deactivation. Furthermore, the network may consider the current environment unsuitable for AI and instruct the terminal to revert to non-AI mode.

[0275] Furthermore, the terminal receives a second message sent by the network, indicating whether or not the terminal can perform inactive AI / ML model evaluation within the measurement interval. If the second message indicates that the terminal can perform inactive AI / ML model evaluation within the measurement interval, then the aforementioned first message does not need to be sent; if the second message indicates that the terminal cannot perform inactive AI / ML model evaluation within the measurement interval, then the aforementioned first message needs to be sent again to indicate where the terminal should perform the evaluation.

[0276] Specifically, when mobility requirements are low, such as when there are few frequency points to be measured that require measurement intervals, the network can be configured to use measurement intervals for inactive AI / ML model evaluation, that is, to exchange the extended measurement time for a small throughput loss caused by AI / ML model evaluation.

[0277] The benefit of introducing the second information is that the network does not need to be configured separately for the timing information used for evaluating inactive AI / ML models, which can reduce signaling overhead and reduce throughput loss.

[0278] Option 2: The terminal receives third information sent by the network. This third information includes a configured period, which, compared to the first information, indicates a duration, such as X milliseconds or Y seconds. The network does not specify the exact location within this duration for the terminal to perform AI / ML model evaluation; this is determined autonomously by the terminal. However, the maximum allowable system loss within this time frame needs to be pre-defined in the protocol. For example, the maximum allowable ACK / NACK loss rate within X milliseconds or Y seconds is Z%.

[0279] The gain from introducing third information is that, compared to the first information, it gives the terminal more degrees of freedom to determine how to perform inactive AI / ML model evaluation. At the same time, considering the throughput loss caused by inactive AI / ML model evaluation, a constraint is imposed by specifying the allowable data loss rate within a predetermined timeframe, achieving a trade-off between degrees of freedom and throughput loss.

[0280] Option 3: The terminal sends a fourth piece of information to the network. This fourth piece of information includes the terminal's desired time for evaluating inactive AI / ML models, such as the period, specific evaluation location, and evaluation duration. The network can refer to this fourth piece of information to configure the first, second, and third pieces of information.

[0281] The gain of the fourth information: Since the network cannot know the terminal's situation, the configuration is not optimal. The terminal can determine how to evaluate based on its own environment and report this information to the network through the fourth information to assist the network in configuration.

[0282] In this disclosure, the terminal receives first information sent by the network. This first information indicates the time information for the terminal to perform inactive model evaluation. Specifically, this time information can be a period of time within a certain duration, during which the terminal only performs inactive AI / ML model evaluation. The first information enables the network and the terminal to have a consistent understanding of when to perform inactive AI / ML model evaluation, which can assist the network in making relevant decisions. For example, if the evaluation is performed at the time specified by the network, switching from a serving AI / ML model to an inactive AI / ML model, even if AI performance degrades, the network can determine that it is due to the AI / ML model evaluation performed within that time period. In this case, the network can accept the AI ​​loss during that period, or not require the terminal to meet the AI ​​performance requirements (which can be understood as the network not needing to take additional measures). If AI / ML performance degrades outside the time specified by the network, the network can consider the current AI / ML model unsuitable, which may trigger model switching, activation, deactivation, or even the network may consider the current environment unsuitable for AI and instruct the terminal to fall back to non-AI mode.

[0283] In this disclosure, the terminal receives second information sent by the network, indicating whether or not the terminal can perform inactive AI / ML model evaluation within a measurement interval. If the second information indicates that the terminal can perform inactive AI / ML model evaluation within the measurement interval, then the aforementioned first information need not be sent; if the second information indicates that the terminal cannot perform inactive AI / ML model evaluation within the measurement interval, then the aforementioned first information needs to be sent to indicate where the evaluation should be performed. The second information enables the network to avoid separately configuring time information for inactive AI / ML model evaluation, thereby reducing signaling overhead and minimizing losses caused by inactive model evaluation.

[0284] In this disclosure, the terminal receives third information sent by the network. This third information includes a configured period and the maximum allowable loss from inactive model evaluation within a certain timeframe. Compared to the first information, the third information only provides a duration, such as X milliseconds or Y seconds. The network does not specify the exact location where the terminal performs AI / ML model evaluation within this duration; this is determined autonomously by the terminal. However, it is necessary to specify the maximum allowable system loss within this timeframe (e.g., the maximum allowable ACK / NACK loss rate of Z% within X milliseconds or Y seconds). The loss from the terminal's inactive model evaluation cannot exceed the maximum value in the third information. Compared to the first information, the third information gives the terminal more freedom to determine how to perform inactive AI / ML model evaluation. Simultaneously, considering the performance loss caused by inactive AI / ML model evaluation, a trade-off between freedom and performance loss is achieved by specifying the maximum allowable loss rate within the agreed duration.

[0285] In this disclosure, the terminal sends a fourth piece of information to the network. This fourth piece of information includes the terminal's desired time information for evaluating inactive AI / ML models, such as the period, specific evaluation location, and evaluation duration. The network can refer to this fourth piece of information to configure the first, second, and third pieces of information.

[0286] Evaluating the performance of the inactive model requires interrupting the currently applied AI / ML model and switching to the inactive model for evaluation. This switching and evaluation process leads to an interruption of the AI ​​service. Furthermore, if the endpoint arbitrarily decides when to perform the inactive model evaluation, when AI / ML performance degrades, the network cannot determine whether the problem is due to a mismatch between the current AI / ML model and the endpoint switching to the inactive model. This could lead to incorrect network decisions and a decrease in system performance.

[0287] To address the aforementioned issues, this disclosure proposes a solution, specifically: The terminal receives first information sent by the network. This first information includes time information for the terminal to perform inactive model evaluation. Specifically, this time information can be a period of time within a certain duration, during which the terminal only performs inactive AI / ML model evaluation (it can also include the conversion time from the currently served AI / ML model to the inactive AI / ML model, or the conversion time from the inactive AI / ML model to the original service AI / ML model). Further, the terminal receives second information sent by the network, indicating whether / whether the terminal can perform inactive AI / ML model evaluation within a measurement interval. Optionally, the terminal receives third information configured by the network. This third information only includes the configured period, such as X milliseconds or Y seconds, and the maximum allowed ACK / NACK loss rate within this time range is pre-defined in the protocol. The terminal is allowed to send fourth information to the network, which includes the terminal's desired time information for performing inactive AI / ML model evaluation, including the period, specific evaluation location, and evaluation duration.

[0288] To implement the method of this disclosure embodiment, this disclosure embodiment also provides a model and / or functional evaluation device 300, disposed on a first device, as shown in FIG3, FIG3 being a schematic structural diagram of a model and / or functional evaluation device according to an embodiment of this disclosure; the device 300 includes at least one of the following:

[0289] Evaluation unit 301 is used to perform model and / or functional evaluation in the first instance;

[0290] Performance unit 302 is used to determine whether the performance loss is less than or equal to a first threshold.

[0291] In one embodiment, the model includes one of the following:

[0292] Deactivate the model (inactive model);

[0293] Artificial intelligence (AI) and / or machine learning (ML) models;

[0294] Deactivated AI and / or ML models; and / or,

[0295] The function includes one of the following:

[0296] Deactivate functionality;

[0297] Artificial intelligence (AI) and / or machine learning (ML) capabilities;

[0298] Deactivate AI and / or ML functionality.

[0299] In one embodiment, the first time includes at least one of the following: the time for the service's model to transition to an inactive model, the model's evaluation time, and the time for the inactive model to transition back to the service's model; and / or,

[0300] The first time includes at least one of the following: the time when the service's functionality is converted to inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality is converted to the service's functionality.

[0301] In one embodiment, the device 300 further includes a receiving unit for receiving a first message; the first message indicating at least one of the following:

[0302] The period of the first time;

[0303] The offset of the first time;

[0304] The length of the first time period.

[0305] In one embodiment, the first time period is constituted by at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and / or functional evaluation; the first time unit is any one of the at least one time units; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and / or a second value; the first value indicates that the first time unit is used for model and / or functional evaluation; the second value indicates that the first time unit is not used for model and / or functional evaluation.

[0306] In one embodiment, a second time is constituted by at least one first time, and bit information is used to indicate whether or not a model and / or function evaluation is performed during the second time; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and / or a second value; the first value indicates that the first time is used for model and / or function evaluation; the second value indicates that the first time is not used for model and / or function evaluation.

[0307] In one embodiment, the device 300 further includes a transmitting unit for transmitting second information; the second information is used to instruct the first device to complete the evaluation of the model; and / or, the second information is used to instruct the first device to complete the evaluation of the function.

[0308] In one embodiment, the second information includes at least one of the following:

[0309] Instructions for completing the assessment;

[0310] Model Index ID;

[0311] Function Index ID.

[0312] In one embodiment, the receiving unit is further configured to receive third information;

[0313] The third information includes at least one of the following:

[0314] Instruct the first device to evaluate the model within the measurement interval;

[0315] Instruct the first device to perform a functionality evaluation within the measurement interval;

[0316] Instruct the first device not to evaluate the model during the measurement interval;

[0317] This instructs the first device not to perform a functionality evaluation during the measurement interval.

[0318] In one embodiment, determining that the performance loss is less than or equal to a first threshold includes one of the following:

[0319] The loss rate of the ACK and / or NACK characters is less than or equal to the first threshold;

[0320] The loss rate of ACK and / or NACK characters within the third time period is less than or equal to the first threshold.

[0321] In one embodiment, the sending unit is further configured to send fourth information; the fourth information includes fourth time information of the first device performing model and / or functional evaluation;

[0322] The fourth time information includes at least one of the following:

[0323] The cycle of the model and / or functional evaluation;

[0324] The offset of the model and / or functional evaluation;

[0325] The duration of the model and / or functional evaluation.

[0326] To implement the method on the second device side of this disclosure embodiment, this disclosure embodiment also provides a model and / or function evaluation device, disposed on the second device, as shown in FIG4. FIG4 is a structural schematic diagram of another model and / or function evaluation device according to this disclosure embodiment. The device 400 includes:

[0327] Sending unit 401 is configured to send a first message to a first device; the first message indicates at least one of the following:

[0328] The first time period;

[0329] The offset of the first time;

[0330] The length of the first time period.

[0331] In one embodiment, the first time includes at least one of the following: the time for the service's model to transition to an inactive model, the model's evaluation time, and the time for the inactive model to transition back to the service's model; and / or,

[0332] The first time includes at least one of the following: the time when the service's functionality is converted to inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality is converted to the service's functionality.

[0333] In one embodiment, the first time period is constituted by at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and / or functional evaluation; the first time unit is any one of the at least one time units; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and / or a second value; the first value indicates that the first time unit is used for model and / or functional evaluation; the second value indicates that the first time unit is not used for model and / or functional evaluation.

[0334] In one embodiment, a second time is constituted by at least one first time, and bit information is used to indicate whether or not a model and / or function evaluation is performed during the second time; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and / or a second value; the first value indicates that the first time is used for model and / or function evaluation; the second value indicates that the first time is not used for model and / or function evaluation.

[0335] In one embodiment, the device 400 further includes a receiving unit for receiving second information sent by the first device; the second information is used to instruct the first device to complete the evaluation of the model; and / or, the second information is used to instruct the first device to complete the evaluation of the function.

[0336] In one embodiment, the second information includes at least one of the following:

[0337] Instructions for completing the assessment;

[0338] Model Identifier ID;

[0339] Functional Identifier ID.

[0340] In one embodiment, the sending unit 401 is used to send third information to the first device;

[0341] The third information includes at least one of the following:

[0342] Instruct the first device to evaluate the model within the measurement interval;

[0343] Instruct the first device to perform a functionality evaluation within the measurement interval;

[0344] Instruct the first device not to evaluate the model during the measurement interval;

[0345] This instructs the first device not to perform a functionality evaluation during the measurement interval.

[0346] In one embodiment, the receiving unit is further configured to receive fourth information sent by the first device; the fourth information includes fourth time information of the first device performing model and / or functional evaluation;

[0347] The fourth time information includes at least one of the following:

[0348] The cycle of the model and / or functional evaluation;

[0349] The offset of the model and / or functional evaluation;

[0350] The duration of the model and / or functional evaluation.

[0351] It should be noted that the model and / or function evaluation apparatus provided in the above embodiments is only illustrated by the division of the above program modules when performing model and / or function evaluation. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the model and / or function evaluation apparatus provided in the above embodiments and the model and / or function evaluation method embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0352] Based on the hardware implementation of the above program modules, this disclosure also provides a model and / or functional evaluation device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of any of the methods described in the first device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side.

[0353] Correspondingly, embodiments of this disclosure provide a computer program product, including a computer program, on which the computer program is stored, wherein when the computer program is executed by a processor, it implements the steps of any of the methods described in the first device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side.

[0354] Correspondingly, embodiments of this disclosure provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described in the first device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side.

[0355] It should be noted that the descriptions of the storage medium and device embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0356] It should be noted that the model and / or functional evaluation device can be a first device; Figure 5 is a schematic diagram of the structure of the first device according to an embodiment of the present disclosure. As shown in Figure 5, the first device 500 includes: a first processor 501 and a first memory 503. Optionally, the first device 500 may also include a first communication interface 502.

[0357] It is understood that the first memory 503 can be volatile memory or non-volatile memory, or both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The first memory 503 described in the embodiments of this disclosure is intended to include, but is not limited to, these and any other suitable types of memory.

[0358] The methods disclosed in the above embodiments of this disclosure can be applied to, or implemented by, the first processor 501. The first processor 501 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware or by instructions in the form of software within the first processor 501. The first processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The first processor 501 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically in the first memory 503. The first processor 501 reads information from the first memory 503 and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0359] It should be noted that the model and / or functional evaluation device can be a second device; Figure 6 is a schematic diagram of the structure of the second device in this embodiment of the present disclosure. As shown in Figure 6, the second device 600 includes: a second processor 601 and a second memory 603. Optionally, the second device 600 may also include a second communication interface 602.

[0360] It is understood that the second memory 603 can be volatile memory or non-volatile memory, or both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The second memory 603 described in the embodiments of this disclosure is intended to include, but is not limited to, these and any other suitable types of memory.

[0361] The methods disclosed in the above embodiments of this disclosure can be applied to, or implemented by, the second processor 601. The second processor 601 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware or by instructions in the form of software within the second processor 601. The second processor 601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The second processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically a second memory 603. The second processor 601 reads information from the second memory 603 and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0362] To implement the method provided in this disclosure, this disclosure also provides a model and / or function evaluation system, as shown in FIG7. FIG7 is a schematic diagram of the structure of the model and / or function evaluation system of this disclosure, which includes: a first device 701 and a second device 702.

[0363] It should be noted that the specific processing procedures of the first device 701 and the second device 702 have been detailed above and will not be repeated here.

[0364] In an exemplary embodiment, the device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0365] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this disclosure, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above-described embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0366] It should be noted that, in this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0367] The methods disclosed in the several method embodiments provided in this disclosure can be arbitrarily combined without conflict to obtain new method embodiments.

[0368] The features disclosed in the several product embodiments provided in this disclosure can be combined arbitrarily without conflict to obtain new product embodiments.

[0369] The features disclosed in the several method or device embodiments provided in this disclosure can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0370] The above description is merely an embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

[0371] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0372] Furthermore, the technical solutions described in the embodiments of this disclosure can be combined arbitrarily without conflict.

[0373] The above description is merely a preferred embodiment of this disclosure and is not intended to limit the scope of protection of this disclosure.

Claims

1. A model and / or functional evaluation method, performed by a first device, said method comprising at least one of the following: Conduct model and / or functional evaluations as soon as possible; Determine that the performance loss is less than or equal to the first threshold.

2. The method according to claim 1, wherein, The model includes one of the following: Deactivate the model (inactive model); Artificial intelligence (AI) and / or machine learning (ML) models; Deactivated AI and / or ML models; and / or, The function includes one of the following: Deactivate functionality; Artificial intelligence (AI) and / or machine learning (ML) capabilities; Deactivate AI and / or ML functionality.

3. The method according to claim 1 or 2, wherein, The first time includes at least one of the following: the time it takes for the service's model to transition to an inactive model, the time it takes for the model to be evaluated, and the time it takes for the inactive model to transition back to the service's model; and / or, The first time includes at least one of the following: the time when the service's functionality is converted to inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality is converted to the service's functionality.

4. The method according to claim 1, wherein, The method further includes: Receive a first message; the first message indicates at least one of the following: The period of the first time; The offset of the first time; The length of the first time period.

5. The method according to claim 1 or 2, wherein, The first time period is composed of at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and / or function evaluation; the first time unit is any time unit among the at least one time unit; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and / or a second value; the first value indicates that the first time unit is used for model and / or function evaluation; the second value indicates that the first time unit is not used for model and / or function evaluation.

6. The method according to claim 1 or 2, wherein, The second time is constituted by at least one first time, and bit information is used to indicate whether the first time in the second time is used for model and / or function evaluation; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and / or a second value; the first value indicates that the first time is used for model and / or function evaluation; the second value indicates that the first time is not used for model and / or function evaluation.

7. The method according to claim 1 or 2, wherein, The method further includes: Send a second message; the second message is used to instruct the first device to complete the evaluation of the model; and / or, the second message is used to instruct the first device to complete the evaluation of the function.

8. The method according to claim 7, wherein, The second information includes at least one of the following: Instructions for completing the assessment; Model Index ID; Function Index ID.

9. The method according to claim 1, wherein, The method further includes: Receive third-party information; The third information includes at least one of the following: Instruct the first device to evaluate the model within the measurement interval; Instruct the first device to perform a functionality evaluation within the measurement interval; Instruct the first device not to evaluate the model during the measurement interval; This instructs the first device not to perform a functionality evaluation during the measurement interval.

10. The method according to claim 1, wherein, The determination that the performance loss is less than or equal to the first threshold includes one of the following: The loss rate of the ACK and / or NACK characters is less than or equal to the first threshold; The loss rate of ACK and / or NACK characters within the third time period is less than or equal to the first threshold.

11. The method according to claim 1, wherein, The method further includes: Send a fourth message; the fourth message includes fourth time information of the first device performing model and / or functional evaluation; The fourth time information includes at least one of the following: The cycle of the model and / or functional evaluation; The offset of the model and / or functional evaluation; The duration of the model and / or functional evaluation.

12. A model and / or functional evaluation method, performed by a second device, the method comprising: Send a first message to the first device; the first message indicates at least one of the following: The first time period; The offset of the first time; The length of the first time period.

13. The method according to claim 12, wherein, The first time includes at least one of the following: the time it takes for the service's model to transition to an inactive model, the time it takes for the model to be evaluated, and the time it takes for the inactive model to transition back to the service's model; and / or, The first time includes at least one of the following: the time when the service's functionality is converted to inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality is converted to the service's functionality.

14. The method according to claim 12, wherein, The first time period is composed of at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and / or function evaluation; the first time unit is any time unit among the at least one time unit; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and / or a second value; the first value indicates that the first time unit is used for model and / or function evaluation; the second value indicates that the first time unit is not used for model and / or function evaluation.

15. The method according to claim 12, wherein, The second time is constituted by at least one first time, and bit information is used to indicate whether the first time in the second time is used for model and / or function evaluation; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and / or a second value; the first value indicates that the first time is used for model and / or function evaluation; the second value indicates that the first time is not used for model and / or function evaluation.

16. The method according to claim 12, wherein, The method further includes: The system receives second information sent by the first device; the second information is used to instruct the first device to complete the evaluation of the model; and / or, the second information is used to instruct the first device to complete the evaluation of the function.

17. The method according to claim 16, wherein, The second information includes at least one of the following: Instructions for completing the assessment; Model Identifier ID; Functional Identifier ID.

18. The method according to claim 12, wherein, The method further includes: Send the third information to the first device; The third information includes at least one of the following: Instruct the first device to evaluate the model within the measurement interval; Instruct the first device to perform a functionality evaluation within the measurement interval; Instruct the first device not to evaluate the model during the measurement interval; This instructs the first device not to perform a functionality evaluation during the measurement interval.

19. The method according to claim 12, wherein, The method further includes: Receive fourth information sent by the first device; the fourth information includes fourth time information of the first device performing model and / or functional evaluation; The fourth time information includes at least one of the following: The cycle of the model and / or functional evaluation; The offset of the model and / or functional evaluation; The duration of the model and / or functional evaluation.

20. A model and / or functional evaluation apparatus, disposed in a first device, the apparatus comprising at least one of the following: An evaluation unit is used to perform model and / or functional evaluation in the first instance. Performance unit, used to determine if the performance loss is less than or equal to a first threshold.

21. A model and / or functional evaluation apparatus, disposed in a second device, the apparatus comprising: The sending unit is configured to send a first message to the first device; the first message indicates at least one of the following: The first time period; The offset of the first time; The length of the first time period.

22. A model and / or functional evaluation device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 11; or, when the processor executes the program, it implements the steps of the method according to any one of claims 12 to 19.

23. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11; or, when the computer program is executed by a processor, it implements the steps of the method according to claims 12 to 19.

24. A storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11, or the steps of the method according to claims 12 to 19.

Images