Wireless communication method and communication equipment

Abstract

The invention provides a wireless communication method and communication equipment. The method comprises the following steps: a first device receives first information sent by a second device, wherein the first information is used for carrying out performance test and / or performance evaluation on a first model which is not activated and used in the first device. In the embodiment of the invention, a mode (including performance testing and / or performance evaluation) for testing the first model is provided, and the first model can be understood as a model which is not activated for use, or the first model is a model which needs to be tested (including performance testing and / or performance evaluation) but is not officially used. Namely, in the embodiment of the invention, the first model which is not activated and used can be tested in advance, and compared with the traditional scheme that only the model which is activated and used (or formally used) can be monitored, the method is beneficial to reducing the probability of communication quality reduction caused by blindly activating an inapplicable model.

Description

Wireless communication method and communication device Technical Field

[0001] The present application relates to the field of communication technology, and more specifically, to a wireless communication method and communication device. Background Art

[0002] The current model monitoring process mainly monitors the activated model. However, this mechanism for monitoring the activated model will directly lead to a real-time decline in user communication performance once the activated model performs poorly. For example, the first device may activate and use a model that is not suitable in the first place, and after a period of model monitoring, it is discovered that the activated model is not applicable, resulting in low communication quality based on the model during the period of model monitoring. For example, the throughput based on the model decreases during the period of model monitoring, and for another example, the bit error rate of the data transmitted based on the model increases during the period of model monitoring.

[0003] Summary of the Invention

[0004] The present application provides a wireless communication method and a communication device. The following introduces various aspects of the present application.

[0005] In a first aspect, a method for wireless communication is provided, including: a first device receiving first information sent by a second device, wherein the first information is used to perform performance testing and / or performance evaluation on a first model that is not activated and used in the first device.

[0006] According to a second aspect, a method for wireless communication is provided, comprising: a second device sending first information to a first device, wherein the first information is used to perform performance testing and / or performance evaluation on a first model that is not activated for use by the first device.

[0007] According to a third aspect, a communication device is provided, which is a first device and includes: a receiving unit for receiving first information sent by a second device, wherein the first information is used to perform performance testing and / or performance evaluation on a first model that is not activated and used in the first device.

[0008] In a fourth aspect, a communication device is provided, which is a second device and includes: a sending unit for sending first information to a first device, wherein the first information is used to perform performance testing and / or performance evaluation on a first model that is not activated for use by the first device.

[0009] In a fifth aspect, a communication device is provided, comprising a processor, a memory, and a communication interface, wherein the memory is used to store one or more computer programs, and the processor is used to call the computer program in the memory so that the terminal device executes part or all of the steps in the methods of the above aspects.

[0010] In a sixth aspect, an embodiment of the present application provides a communication system, which includes the first device and / or the second device described above. In another possible design, the system may also include other devices that interact with the first device and / or the second device in the solution provided in the embodiment of the present application.

[0011] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program enables a communication device (for example, a first device or a second device) to perform some or all of the steps in the methods of the above aspects.

[0012] In an eighth aspect, embodiments of the present application provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a communication device (e.g., a first device and / or a second device) to perform some or all of the steps of the methods described in each of the above aspects. In some implementations, the computer program product may be a software installation package.

[0013] In a ninth aspect, an embodiment of the present application provides a chip comprising a memory and a processor, wherein the processor can call and run a computer program from the memory to implement some or all of the steps described in the methods of the above aspects.

[0014] In an embodiment of the present application, a method for testing a first model (including performance testing and / or performance evaluation) is provided, wherein the first model can be understood as a model that is not activated for use, or in other words, the first model is a model that needs to be tested (including performance testing and / or performance evaluation) but is not formally used. In other words, in an embodiment of the present application, the first model that is not activated for use can be tested in advance. Compared with the traditional solution that can only monitor the model that is activated for use (or formally used), this helps to reduce the probability of blindly activating an inappropriate model and causing a decrease in communication quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG1 is a wireless communication system 100 used in an embodiment of the present application.

[0016] FIG2 is a schematic flowchart of a wireless communication method according to an embodiment of the present application.

[0017] FIG3 is a schematic diagram of a communication device according to an embodiment of the present application.

[0018] FIG4 is a schematic diagram of a communication device according to another embodiment of the present application.

[0019] FIG5 is a schematic structural diagram of a communication device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] The technical solution in this application will be described below with reference to the accompanying drawings.

[0021] Figure 1 illustrates a wireless communication system 100 used in an embodiment of the present application. The wireless communication system 100 may include a network device 110 and a terminal device 120. The network device 110 may be a device that communicates with the terminal device 120. The network device 110 may provide communication coverage for a specific geographic area and may communicate with the terminal device 120 within the coverage area.

[0022] In some implementations, the terminal devices 120 may communicate directly with each other, where the communication link may be referred to as a side link.

[0023] FIG1 exemplarily shows a network device and two terminals. Optionally, the wireless communication system 100 may include multiple network devices and each network device may include other numbers of terminal devices within its coverage area, which is not limited in the embodiments of the present application.

[0024] Optionally, the wireless communication system 100 may further include other network entities such as a network controller and a mobility management entity, which is not limited in the embodiment of the present application.

[0025] It should be understood that the technical solutions of the embodiments of the present application can be applied to various communication systems, such as: fifth generation (5G) system or new radio (NR), long term evolution (LTE) system, LTE frequency division duplex (FDD) system, LTE time division duplex (TDD), etc. The technical solutions provided in this application can also be applied to future communication systems, such as the sixth generation mobile communication system, satellite communication system, etc.

[0026] The terminal device in the embodiments of the present application may also be referred to as user equipment (UE), access terminal, user unit, user station, mobile station, mobile station (MS), mobile terminal (MT), remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device. The terminal device in the embodiments of the present application may refer to a device that provides voice and / or data connectivity to a user and can be used to connect people, objects and machines, such as a handheld device with wireless connection function, a vehicle-mounted device, etc. The terminal device in the embodiments of the present application can be a mobile phone, a tablet computer, a laptop computer, a PDA, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc. Optionally, the UE can be used to act as a base station. For example, the UE can act as a scheduling entity that provides sidelink signals between UEs in V2X or D2D, etc. For example, a cellular phone and a car communicate with each other using sidelink signals. The cellular phone and smart home devices communicate without relaying the communication signal through the base station.

[0027] The network device in the embodiments of the present application may be a device for communicating with a terminal device, and may also be referred to as an access network device or a radio access network device. For example, the network device may be a base station. The network device in the embodiments of the present application may refer to a radio access network (RAN) node (or device) that connects a terminal device to a wireless network. A base station can broadly cover various names as follows, or be replaced with the following names, such as: NodeB, evolved NodeB (eNB), next generation NodeB (gNB), relay station, access point, transmission point (TRP), transmission point (TP), master station MeNB, secondary station SeNB, multi-standard radio (MSR) node, home base station, network controller, access node, wireless node, access point (AP), transmission node, transceiver node, baseband unit (BBU), remote radio unit (RRU), active antenna unit (AAU), remote radio head (RRH), central unit (CU), distributed unit (DU), positioning node, etc. A base station can be a macro base station, a micro base station, a relay node, a donor node or the like, or a combination thereof. A base station can also refer to a communication module, a modem or a chip used to be set in the aforementioned device or apparatus. The base station can also be a mobile switching center and a device that performs base station functions in device-to-device D2D, vehicle-to-everything (V2X), and machine-to-machine (M2M) communications, a network-side device in a 6G network, or a device that performs base station functions in future communication systems. The base station can support networks with the same or different access technologies. The embodiments of this application do not limit the specific technology and specific device form used by the network equipment.

[0028] Base stations can be fixed or mobile. For example, a helicopter or drone can be configured to act as a mobile base station, and one or more cells can move based on the location of the mobile base station. In other examples, a helicopter or drone can be configured to act as a device that communicates with another base station.

[0029] In some deployments, the network device in the embodiments of the present application may refer to a CU or a DU, or the network device may include a CU and a DU. The gNB may also include an AAU.

[0030] The network equipment and terminal devices can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; they can also be deployed on water; they can also be deployed in the air on aircraft, balloons, and satellites. The embodiments of this application do not limit the scenarios in which the network equipment and terminal devices are located.

[0031] It should be understood that all or part of the functions of the communication device in this application can also be implemented through software functions running on hardware, or through virtualization functions instantiated on a platform (such as a cloud platform).

[0032] Model monitoring process

[0033] With the development of technology, models have been introduced into communication systems. For example, signal encoding and / or decoding are performed based on models. Another example is beam selection based on models. Another example is data adjustment and / or demodulation based on models. However, due to changes in factors such as the environment and configuration, the model may not work properly, resulting in the inability to communicate based on the model. Currently, in order to monitor the working status of the model and avoid communication failures or performance degradation caused by poor model performance, a model monitoring process has been introduced. During this process, the model's proper functioning can be monitored and the monitoring results can be sent to decision-makers so that they can make management decisions based on the model's monitoring results.

[0034] Based on the model monitoring process described above, it can be seen that the current model monitoring process mainly monitors the activated model. However, this mechanism for monitoring the activated model will directly lead to a real-time decline in user communication performance once the activated model has poor performance. For example, the first device may activate and use a model that is not suitable in the first place, and after a period of model monitoring, it is discovered that the activated model is not applicable, resulting in low communication quality based on the model during the period of model monitoring. For example, the throughput based on the model decreases during the period of model monitoring. For another example, the bit error rate of the data transmitted based on the model increases during the period of model monitoring.

[0035] Therefore, in response to the above problems, an embodiment of the present application provides a method for testing the first model (including performance testing and / or performance evaluation), wherein the first model can be understood as a model that is not activated for use, or in other words, the first model is a model that needs to be tested (including performance testing and / or performance evaluation) but is not formally used. That is to say, in an embodiment of the present application, the first model that is not activated for use can be tested in advance. Compared with the traditional solution that can only monitor the model that is activated for use (or formally used), it helps to reduce the probability of blindly activating an inapplicable model and causing a decrease in communication quality.

[0036] On the other hand, in an embodiment of the present application, the first model that has not been activated can be tested in advance, which helps to avoid the risk of system performance uncertainty brought about by directly officially running the first model (the so-called formal operation of the model can be understood as using an artificial intelligence (AI) model to replace the corresponding non-AI algorithm) to improve the robustness of the system operation after the communication system introduces the first model.

[0037] It should be noted that the embodiments of the present application do not limit the first model. In some implementations, the first model may be an AI model. For example, the first model may be a model built based on a convolutional neural network (CNN) architecture. For another example, the first model may be a model built based on a recurrent neural network (RNN) architecture. In other implementations, the first model may be a machine learning (ML) model.

[0038] In some implementations, the test on the first model may be performed based on the first information. For ease of understanding, the wireless communication method of the embodiment of the present application is described below with reference to FIG2 . The method shown in FIG2 may include step S210 .

[0039] In step S210, the second device sends first information to the first device, wherein the first information is used to perform performance testing and / or performance evaluation on a first model that is not activated and used in the first device.

[0040] It should be noted that, for ease of description, the above-mentioned performance testing and / or performance evaluation of the first model are collectively referred to as testing the first model. In some implementations, the process of testing the first model may include a process of using the first model for reasoning one or more times. Accordingly, the test results obtained by testing the first model can be determined based on the inference results obtained by using the first model for reasoning one or more times. For example, the test results obtained by testing the first model may include the inference results obtained by using the first model for reasoning one or more times. For another example, the test results obtained by testing the first model may be obtained based on statistics of the inference results obtained by using the first model for reasoning one or more times. The embodiments of the present application are not limited to this.

[0041] In addition, taking the first model as an AI model as an example, the above-mentioned AI model can replace the algorithm of the traditional non-AI model to solve communication problems. In the field of communications, an AI model can replace a single or a group of communication modules composed of non-AI algorithms. Since the functions of the AI ​​model and the algorithm of the traditional non-AI model are similar or identical, when the AI ​​model is officially running, the traditional non-AI communication algorithm is usually deactivated (disabled). In the embodiment of the present application, the first model is different from the aforementioned officially activated (or activated for use) AI model. When testing the first model, although the first model needs to be activated, the corresponding non-AI algorithm does not need to be deactivated. In other words, the test of the first model will not affect the operation of the traditional algorithm. The test process of the first model works independently of the corresponding traditional algorithm. At this time, although the first model is activated for testing, it is a model that is not used formally. Accordingly, in some implementations, after the first device receives the first information, it does not deactivate the traditional non-AI communication algorithm. The test process of the first model associated with the first information works independently of the corresponding traditional algorithm.

[0042] In some implementations, the first information includes one or more of the following: an identifier of the first model; a first parameter; a second parameter; a third parameter; and a fourth parameter.

[0043] Take the example where the first information includes the identifier of the first model, wherein the identifier of the first model can be understood as an identifier associated with the first model that needs to be performance tested but is not used formally.

[0044] It should be noted that the first information may include the identifiers of one or more models. If the first information includes the identifiers of multiple models, the models indicated by the multiple model identifiers are all models that need to be tested, that is, the models indicated by the multiple model identifiers are all models that have not been activated for use, or in other words, the models indicated by the multiple model identifiers are all models that need to be tested (including performance testing and / or performance evaluation) but are not officially used, wherein the multiple model identifiers may include the identifier of the above-mentioned first model.

[0045] For example, the first information includes a first parameter, where the first parameter is used to determine the activation time of the first model. In some implementations, the first parameter is used to configure first delay information, where the first model is activated after a time duration corresponding to the first delay information has elapsed since the first device received the first information. In other words, the first delay information indicates the delay between the time the first information is received and the activation of the first model. Alternatively, the first delay information indicates the time offset between the time the first information is received and the activation of the first model.

[0046] In other implementations, the first parameter is used to configure the absolute time information for activating the first model. In the embodiments of the present application, the presentation method of the absolute time information is not limited. For example, the absolute time information can be represented by UTC time. For another example, the absolute time information can be represented by system frame information, where the system frame information includes one or more of a high frame number, a frame number, a subframe number, a time slot number, and a symbol number.

[0047] Typically, after receiving the first information, although the first device knows that the second device wants to test the first model, the first device needs some time to prepare for testing the first model. For example, the first device needs time to decode the first information. For example, the first device needs time to apply the configuration required for testing the first model. For example, the first device needs time to load the first model algorithm. For example, the first device needs time to load the hardware that matches the first model algorithm. For example, the first device needs time to load the software resources that match the first model. Therefore, in order to reserve preparation time for the first device to test the first model, in an embodiment of the present application, the activation time of the first model can be controlled by a first parameter, which helps to avoid the waste of system resources caused by the second device starting to schedule some resources before the test of the first model is ready.

[0048] For example, if the first parameter is not introduced, the second device may have already sent a reference signal for model testing to the first device before the first model test is ready. At this time, the first device cannot test the first model based on the reference signal because it is not ready. Conversely, if the first parameter is introduced, the second device can send the model test reference signal after the time corresponding to the first parameter has passed since the first information was sent. At this time, the first device is also ready to test the first model, and the first device can test the first model based on the reference signal. This helps to avoid the waste of system resources caused by the second device starting to schedule some resources before the first model test is ready.

[0049] Based on the above introduction to the first parameter, it can be known that in an embodiment of the present application, the above-mentioned first parameter may be related to the capability of the first device, or in other words, the first parameter may be determined based on the capability of the first device. For example, for a first device with higher capability, the activation time of the first model determined by the first parameter may be earlier (for example, the duration corresponding to the first parameter may be shorter). For another example, for a first device with lower capability, the activation time of the first model determined by the first parameter may be later (for example, the duration corresponding to the first parameter may be longer).

[0050] It should be noted that there is an association relationship between the first parameter and the identifier of the model in the first information. In some implementations, the association relationship between the first parameter and the identifier of the model in the first information can be a one-to-one association relationship, wherein the one-to-one association relationship can be understood as a first parameter being associated with the identifier of a model. In other implementations, the association relationship between the first parameter and the identifier of the model in the first information can be a one-to-many association relationship, which helps to reduce the resources required to transmit the first information. Among them, the one-to-many association relationship can be understood as a first parameter being associated with the identifiers of a group of models, or the one-to-many association relationship can be understood as a first parameter being associated with the identifiers of multiple models, or the one-to-many association relationship can be understood as a first parameter being associated with the identifiers of all models in the first information.

[0051] Accordingly, in some implementations, the model associated with the first parameter needs to be activated according to the activation time determined based on the first parameter. Taking the association of the first parameter with the first model as an example, the activation time of the first model is determined based on the first parameter.

[0052] In the embodiments of the present application, there is no limitation on the method for obtaining the first parameter. In some implementations, the first parameter can be carried in the first information and indicated by the second device to the first device, which helps to increase the flexibility of the first parameter configuration. In other implementations, the first parameter can be set in a predefined manner. For example, the first parameter can be set to a default value in the protocol, which helps to reduce the overhead required to transmit the first information.

[0053] It should be emphasized that the above-mentioned methods of obtaining the first parameters can be used in combination with each other. For example, the first information includes the identifiers of multiple models. At this time, the first parameters associated with some models in the multiple models can be carried by the first information, and the first parameters associated with other models in the multiple models can be set in a predefined manner, which helps to reduce the overhead required to transmit the first information while improving the flexibility of configuring the first parameters.

[0054] Taking the example of the first information including the second parameter, the second parameter is used to determine whether the inference result of the first model meets the accuracy requirements. In other words, the second parameter is used to determine whether the difference between the inference result of the first model and the actual result meets the accuracy requirements. In some implementations, the second parameter includes an accuracy threshold, wherein the accuracy threshold is used to determine whether the difference between the inference result of the first model and the corresponding actual result meets the accuracy requirements, or in other words, the accuracy threshold is used to determine whether the difference between a single inference result of the first model and the corresponding actual result meets the accuracy requirements.

[0055] In an embodiment of the present application, by introducing the second parameter, it helps the first device to autonomously determine whether the single inference result of the first model meets the accuracy requirements. Afterwards, the first device can directly perform statistics on the model inference results based on whether the single inference result of the first model meets the accuracy requirements.

[0056] In some implementations, if the absolute value of the difference between the inference result of the first model and the actual result is less than or equal to the accuracy threshold, the inference result of the first model is considered to meet the accuracy requirement. In other implementations, if the absolute value of the difference between the inference result of the first model and the actual result is greater than the accuracy threshold, the inference result of the first model is considered to fail to meet the accuracy requirement.

[0057] It should be noted that there is an association relationship between the second parameter and the identifier of the model in the first information. In some implementations, the association relationship between the second parameter and the identifier of the model in the first information can be a one-to-one association relationship, wherein the one-to-one association relationship can be understood as one second parameter being associated with the identifier of one model. In other implementations, the association relationship between the second parameter and the identifier of the model in the first information can be a one-to-many association relationship, which helps to reduce the resources required to transmit the first information. Among them, the one-to-many association relationship can be understood as one second parameter being associated with the identifiers of a group of models, or the one-to-many association relationship can be understood as one second parameter being associated with the identifiers of multiple models, or the one-to-many association relationship can be understood as one second parameter being associated with the identifiers of all models in the first information.

[0058] In the embodiments of the present application, the actual results mentioned above or later are not limited. In some implementations, the actual results may include any one of actual observation results, actual measurement results, and actual calculation results.

[0059] Taking the example of the first information including the third parameter, the third parameter is used to configure a reference signal for testing the first model, or in other words, the third parameter is used to configure a reference signal resource related to the testing of the first model. Therefore, the third parameter can also be referred to as a reference signal resource configuration. In some implementations, the third parameter can be used to configure one or more of a time domain resource, a code domain resource, a spatial domain resource, and a frequency domain resource of the reference signal.

[0060] In some implementations, the functionality of the first model is associated with the type of reference signal for testing the first model. For example, if the first model is used for CSI compression feedback, the reference signal for testing the first model is a channel state information reference signal (CSI-RS). For another example, if the first model is used for CSI time domain prediction, the reference signal for testing the first model is a CSI-RS. For another example, if the first model is used for beam spatial domain prediction, the reference signal for testing the first model includes a synchronization signal / physical broadcast channel block (SSB) or CSI-RS. For another example, if the first model is used for beam time domain prediction, the reference signal for testing the first model includes an SSB or CSI-RS. For another example, if the first model is used for positioning, the reference signal for testing the first model includes a positioning reference signal (PRS).

[0061] In some scenarios, if the first model can be tested using reference signal resources configured for a traditional algorithm, the first message may not carry the third parameter, thereby reducing the transmission resources required to transmit the first message. In other scenarios, if the first message definition includes the third parameter, but the third parameter is not configured during a configuration process or does not appear in the first message, the first device may default to using the reference signal resources configured for the traditional algorithm to test the first model.

[0062] It should be noted that there is an association relationship between the third parameter and the identifier of the model in the first information. In some implementations, the association relationship between the third parameter and the identifier of the model in the first information can be a one-to-one association relationship, wherein the one-to-one association relationship can be understood as a third parameter being associated with the identifier of a model. In other implementations, the association relationship between the third parameter and the identifier of the model in the first information can be a one-to-many association relationship, which helps to reduce the resources required to transmit the first information. Among them, the one-to-many association relationship can be understood as a third parameter being associated with the identifiers of a group of models, or the one-to-many association relationship can be understood as a third parameter being associated with the identifiers of multiple models, or the one-to-many association relationship can be understood as a third parameter being associated with the identifiers of all models in the first information.

[0063] Taking the example where the first information includes the fourth parameter, wherein the fourth parameter is associated with the test process of the first model, the fourth parameter of the embodiment of the present application is introduced below with reference to Examples 1 to 4.

[0064] Example 1: The fourth parameter is used to configure the running time information of the first timer, wherein the first timer is associated with the inference result of the first model.

[0065] It should be noted that there is an association between the duration information of the first timer and the identifier of the model in the first information. In some implementations, the association between the duration information of the first timer and the identifier of the model in the first information can be a one-to-one association, wherein the one-to-one association can be understood as the association between the duration information of a first timer and the identifier of a model. In other implementations, the association between the duration information of the first timer and the identifier of the model in the first information can be a one-to-many association, which helps to reduce the resources required to transmit the first information. Among them, the one-to-many association can be understood as the association between the duration information of a first timer and the identifier of a group of models, or the one-to-many association can be understood as the association between the duration information of a first timer and the identifiers of multiple models, or the one-to-many association can be understood as the association between the duration information of a first timer and the identifiers of all models in the first information.

[0066] In addition, in the embodiment of the present application, no matter whether the above-mentioned association relationship is a one-to-one association relationship or a one-to-many association relationship, the models indicated by the model identifier in the first information can be associated with a first timer respectively.

[0067] In the embodiment of the present application, there are multiple implementations of the start condition and / or stop condition of the first timer. The following describes the start condition implementation methods 1 and 2 and the stop condition implementation methods 1 and 2 respectively.

[0068] In the first implementation mode of the start-up condition, the start-up condition of the first timer includes that the inference result of the first model does not meet the accuracy requirement, or in other words, the start-up condition of the first timer includes that the single inference result of the first model does not meet the accuracy requirement.

[0069] Of course, in embodiments of the present application, the aforementioned activation conditions may also include other conditions, such as the first timer being in a stopped state. In other words, the aforementioned activation conditions may include the first timer being in a stopped state and the inference result of the first model not meeting the accuracy requirements. For example, the activation conditions for the first timer include: the absolute value of the difference between a single inference result of the first model and the corresponding actual result is greater than the accuracy threshold, and the first timer is in a stopped state.

[0070] Based on the above introduction to the implementation method 1 of the start condition, it can be seen that in an embodiment of the present application, if the first timer is in a stopped state and a certain inference result of the first model meets the accuracy requirements, the first timer will not be started (that is, the first timer continues to be in a stopped state).

[0071] In implementation method 2 of the activation condition, the activation condition for the first timer includes M consecutive inference results of the first model failing to meet the accuracy requirement, where M is a positive integer greater than or equal to 1. If M is a positive integer greater than 1, the activation condition for the first timer can be replaced by multiple consecutive inference results of the first model failing to meet the accuracy requirement.

[0072] Of course, in embodiments of the present application, the aforementioned activation conditions may also include other conditions, such as the first timer being in a stopped state. That is, the aforementioned activation conditions may include the first timer being in a stopped state and the first model's M consecutive inference results failing to meet the accuracy requirement. For example, the activation conditions for the first timer include: the absolute value of the difference between the first model's M consecutive inference results and the corresponding actual results being greater than a precision threshold, and the first timer being in a stopped state.

[0073] It should be noted that, in one implementation, the absolute value of the difference between the M consecutive reasoning results of the above-mentioned first model and the corresponding actual result is greater than the accuracy threshold, which can be understood as the absolute value of the difference between the M consecutive reasoning results of the first model and the corresponding actual result is greater than the accuracy threshold. For example, when the value of M is 2, the absolute value of the difference between the M consecutive reasoning results of the above-mentioned first model and the corresponding actual result is greater than the accuracy threshold, which can include: the absolute value of the difference between the first reasoning result of the first model and the actual result corresponding to the first reasoning result is greater than the accuracy threshold, and the absolute value of the difference between the second reasoning result of the first model and the actual result corresponding to the second reasoning result is greater than the accuracy threshold. In another implementation, in an embodiment of the present application, the absolute value of the difference between the M consecutive reasoning results of the above-mentioned first model and the corresponding actual result is greater than the accuracy threshold, which can be understood as the absolute value of the difference between the average value of the M consecutive reasoning results of the first model and the latest actual result or the average value of the actual results obtained M times is greater than the accuracy threshold. For example, when the value of M is 2, the absolute value of the difference between the M consecutive inference results of the above-mentioned first model and the latest actual result or the average of the actual results obtained M times is greater than the accuracy threshold, which may include: the absolute value of the difference between the first inference result of the first model and the average of the second inference result and the actual result obtained for the second time or the average of the actual results obtained 2 times is greater than the accuracy threshold.

[0074] In addition, based on the above description, it can be seen that the use of M can be related to the inference results of the first model. The embodiments of the present application do not limit the value of M. For example, the value of M can be indicated by the first information. For another example, the value of M can be obtained through one or more of the following information: predefined information; preconfigured information; system broadcast messages; or other dedicated signaling. This is not limited in the embodiments of the present application.

[0075] It should also be noted that there is an association between the value of M and the identifier of the model in the first information. In some implementations, the association between the value of M and the identifier of the model in the first information can be a one-to-one association, wherein the one-to-one association can be understood as one value of M being associated with the identifier of one model. In other implementations, the association between the value of M and the identifier of the model in the first information can be a one-to-many association, which helps to reduce the resources required to transmit the first information. Among them, the one-to-many association can be understood as one value of M being associated with the identifier of a group of models, or the one-to-many association can be understood as one value of M being associated with the identifiers of multiple models, or the one-to-many association can be understood as one value of M being associated with the identifiers of all models in the first information.

[0076] Based on the above introduction to the implementation method 2 of the start condition, it can be seen that in an embodiment of the present application, if the first timer is in a stopped state and the m consecutive inference results of the first model do not meet the accuracy requirements, the first timer will not be started (that is, the first timer continues to be in a stopped state), where m is less than M.

[0077] Stop condition implementation mode 1: the stop condition of the first timer includes that the inference result of the first model meets the accuracy requirement, or in other words, the stop condition of the first timer includes that the single inference result of the first model meets the accuracy requirement.

[0078] Of course, in embodiments of the present application, the above-mentioned stop condition may also include other conditions, for example, the stop condition includes the first timer being in a running state. In other words, the above-mentioned stop condition may include the first timer being in a running state and the inference result of the first model meeting the accuracy requirement. For example, the stop condition of the first timer includes: the absolute value of the difference between a single inference result of the first model and the corresponding actual result is less than or equal to the accuracy threshold, and the first timer is in a running state.

[0079] Based on the above introduction to the stop condition implementation method 1, it can be seen that in an embodiment of the present application, if the first timer is in a running state and any inference result of the first model does not meet the accuracy requirements, the first timer will not be stopped (that is, the first timer continues to run).

[0080] In stop condition implementation method 2, the stop condition for the first timer includes N consecutive inference results of the first model meeting the accuracy requirement, where N is a positive integer greater than or equal to 1. If N is a positive integer greater than 1, the stop condition for the first timer can be replaced by multiple consecutive inference results of the first model meeting the accuracy requirement.

[0081] Of course, in embodiments of the present application, the above-mentioned stop condition may also include other conditions, for example, the stop condition may include that the first timer is in a running state. In other words, the above-mentioned stop condition may include that the first timer is in a running state and that N consecutive inference results of the first model meet the accuracy requirements. For example, the stop condition of the first timer includes: the absolute value of the difference between N consecutive inference results of the first model and the corresponding actual results is less than or equal to the accuracy threshold, and the first timer is in a running state.

[0082] It should be noted that, in one implementation, the absolute value of the difference between the N consecutive inference results of the first model and the corresponding actual results is less than or equal to the accuracy threshold, which can be understood as the absolute value of the difference between the N consecutive inference results of the first model and the corresponding actual results is less than or equal to the accuracy threshold. For example, when the value of N is 2, the absolute value of the difference between the N consecutive inference results of the first model and the corresponding actual results is less than or equal to the accuracy threshold, which can include: the absolute value of the difference between the first inference result of the first model and the actual result corresponding to the first inference result is less than or equal to the accuracy threshold, and the absolute value of the difference between the second inference result of the first model and the actual result corresponding to the second inference result is less than or equal to the accuracy threshold. In another implementation, the absolute value of the difference between the N consecutive inference results of the first model and the corresponding actual results is less than or equal to the accuracy threshold, which can be understood as the absolute value of the difference between the average value of the N consecutive inference results of the first model and the most recently obtained actual result is less than or equal to the accuracy threshold. For example, when the value of N is 2, the absolute value of the difference between the N consecutive reasoning results of the above-mentioned first model and the latest actual result obtained is less than or equal to the precision threshold, which may include: the absolute value of the difference between the average value of the first reasoning result and the second reasoning result of the first model and the actual result obtained for the second time is less than or equal to the precision threshold. In another implementation, the absolute value of the difference between the N consecutive reasoning results of the above-mentioned first model and the corresponding actual result is less than or equal to the precision threshold, which can be understood as the absolute value of the difference between the average value of the N consecutive reasoning results of the first model and the average value of the actual results obtained N times is less than or equal to the precision threshold. For example, when the value of N is 2, the absolute value of the difference between the N consecutive reasoning results of the above-mentioned first model and the average value of the actual results obtained N times is less than or equal to the precision threshold, which may include: the absolute value of the difference between the average value of the first reasoning result and the second reasoning result of the first model and the average value of the actual results obtained 2 times is less than or equal to the precision threshold.

[0083] In addition, based on the above description, it can be seen that the use of N can be related to the inference results of the first model. The embodiments of the present application do not limit the value of N. For example, the value of N can be indicated by the first information. For another example, the value of N can be obtained through one or more of the following information: predefined information; preconfigured information; system broadcast messages; or other dedicated signaling. This is not limited in the embodiments of the present application.

[0084] It should also be noted that there is an association between the value of N and the identifier of the model in the first information. In some implementations, the association between the value of N and the identifier of the model in the first information can be a one-to-one association, wherein the one-to-one association can be understood as one value of N being associated with the identifier of one model. In other implementations, the association between the value of N and the identifier of the model in the first information can be a one-to-many association, which helps to reduce the resources required to transmit the first information. Among them, the one-to-many association can be understood as one value of N being associated with the identifier of a group of models, or the one-to-many association can be understood as one value of N being associated with the identifiers of multiple models, or the one-to-many association can be understood as one value of N being associated with the identifiers of all models in the first information.

[0085] Based on the above introduction to the stop condition implementation method 2, it can be seen that in an embodiment of the present application, if the first timer is in a running state and the consecutive n inference results of the first model meet the accuracy requirements, the first timer will not be stopped (that is, the first timer continues to be in a running state), where n is less than N.

[0086] It should be noted that, in the embodiments of the present application, the start condition and stop condition of the first timer can be used separately or in combination. For example, the start condition implementation method 1 can be used in combination with the stop condition implementation method 1. For another example, the start condition implementation method 2 can be used in combination with the stop condition implementation method 2.

[0087] Example 2: The fourth parameter is used to configure the running time information of the second timer. The second timer is associated with the test result reporting operation of the first model, or in other words, the test result reporting operation of the first model is triggered based on the second timer. For example, the test result reporting operation of the first model is triggered based on the expiration of the second timer, or in other words, in response to the expiration of the second timer, the first device can send the test result of the first model to the second device.

[0088] In some implementations, the start time of the second timer is determined based on the time when the first device receives the first information. For example, the start time of the second timer may be the time when the first device receives the first information. In another example, the start time of the second timer may be offset by a first offset value from the time when the first device receives the first information, where the first offset value is predefined, preconfigured, or determined by network device configuration.

[0089] In some other implementations, the start time of the second timer is determined based on the time when the first device activates the first model (also referred to as the "activation time of the first model"). For example, the start time of the second timer can be the activation time of the first model. In another example, the start time of the second timer can be offset from the activation time of the first model by a second offset value, where the second offset value is predefined, preconfigured, or determined by network device configuration.

[0090] It should be noted that there is an association between the duration information of the second timer and the identifier of the model in the first information. In some implementations, the association between the duration information of the second timer and the identifier of the model in the first information can be a one-to-one association, wherein the one-to-one association can be understood as the association between the duration information of a second timer and the identifier of a model. In other implementations, the association between the duration information of the second timer and the identifier of the model in the first information can be a one-to-many association, which helps to reduce the resources required to transmit the first information. Among them, the one-to-many association can be understood as the association between the duration information of a second timer and the identifier of a group of models, or the one-to-many association can be understood as the association between the duration information of a second timer and the identifiers of multiple models, or the one-to-many association can be understood as the association between the duration information of a second timer and the identifiers of all models in the first information.

[0091] Example 3: The fourth parameter is used to configure a target inference count threshold for testing the first model.

[0092] In some implementations, the use of the target inference number threshold is related to the inference result of the first model. The following will be specifically introduced in conjunction with implementations 1 to 5. For the sake of brevity, it will not be repeated here.

[0093] It should be noted that there is an association between the target inference number threshold and the identifier of the model in the first information. In some implementations, the association between the target inference number threshold and the identifier of the model in the first information can be a one-to-one association, wherein the one-to-one association can be understood as one target inference number threshold being associated with the identifier of one model. In other implementations, the association between the target inference number threshold and the identifier of the model in the first information can be a one-to-many association, which helps to reduce the resources required to transmit the first information. Among them, the one-to-many association can be understood as one target inference number threshold being associated with the identifiers of a group of models, or the one-to-many association can be understood as one target inference number threshold being associated with the identifiers of multiple models, or the one-to-many association can be understood as one target inference number threshold being associated with the identifiers of all models in the first information.

[0094] The above introduces several examples of the fourth parameter in the embodiments of the present application. In some implementations, the above examples of the fourth parameter can be used independently. In other implementations, the above examples of the fourth parameter can be used in combination with each other. For example, Example 1 can be used in combination with Example 2, that is, the fourth parameter is used to configure the duration information of the first timer and the duration information of the second timer. For another example, Example 3 can be used in combination with Example 2, that is, the fourth parameter is used to configure the duration information of the second timer and the target inference number threshold.

[0095] For ease of understanding, the following introduces the implementation method of the test-related information of the first model recorded by the first device in the embodiment of the present application and the implementation method of transmitting the second information based on several examples of the fourth parameter in the embodiment of the present application.

[0096] Implementation method 1: Assume that the fourth parameter is used to configure the running time information of the first timer (ie, Example 1 above).

[0097] In some implementations, the method further includes: when the first timer times out, the first device may send second information to the second device, and the second device accordingly receives the second information sent by the first device. The second information includes an identifier of the first model associated with the first timer that timed out, and the identifier of the first model associated with the first timer indicates that the first timer associated with the corresponding model has timed out.

[0098] In the embodiment of the present application, when the first timer times out, it can be understood that the first model does not meet the accuracy requirements. At this time, the first device can promptly inform the second device through the second information.

[0099] In some implementations, the method further includes: when the first timer times out, the first device stops running the first model associated with the timed-out first timer. Of course, in the embodiment of the present application, when the first timer times out, the first device may also continue to test the first model.

[0100] Implementation method 2: Assume that the fourth parameter is used to configure the running time information of the first timer and the running time information of the second timer (ie, a combination of Example 1 and Example 2 above).

[0101] In some implementations, the above method also includes: during the operation of the second timer, when the first timer times out, the first device records the identifier of the first model associated with the first timer that times out, and stops running the first model associated with the first timer that times out; and / or, when the second timer times out, the first device sends second information to the second device, and accordingly, the second device receives the second information sent by the first device, the second information includes the identifier of the first model associated with the first timer that times out, and the identifier of the first model associated with the first timer indicates that the first timer associated with the corresponding model has timed out.

[0102] In an embodiment of the present application, if the first timer times out, the first device can first record the identifier of the first model associated with the first timer, stop running the first model, and then send the second information to the second device after the second timer times out. In other words, the first device does not need to immediately send the second information when the first timer associated with any first model times out. It only needs to temporarily record the identifier of the corresponding first model and wait until the second timer times out before sending the identifier information of one or more first models associated with the timed-out first timer together with the second information, thereby avoiding the overhead required to frequently transmit the second information.

[0103] For example, the second timer runs for 60 seconds, and during this time, Models A, B, and C need to be tested. Assume that after the first device receives the first message, the second timer begins running. Ten seconds into the second timer's run, the first timer associated with Model A expires. At this point, the first device can stop running Model A and record Model A's identifier. Then, 30 seconds into the second timer's run, the first timer associated with Model B expires. At this point, the first device can stop running Model B and record Model B's identifier. Then, 50 seconds into the second timer's run, the first timer associated with Model C expires. At this point, the first device stops running Model C and records Model C's identifier. When the second timer expires (i.e., 60 seconds into the second timer's run), the first device reports a second message to the second device, including the identifiers of Models A, B, and C. In other words, the identifiers of Models A, B, and C can be transmitted all at once in the second message, helping to reduce the resources required to transmit the second message.

[0104] As described above, the second information may include the identifier of the first model associated with the first timer that timed out. Therefore, during the operation of the second timer, the first device may record the identifier of the first model associated with the first timer that timed out for subsequent transmission via the second information. For example, if the first timer associated with the first model times out, the first device may record the identifier of the first model as a new information entry in the cache.

[0105] In the embodiment of the present application, the second information is not specifically limited. In some implementations, the above-mentioned second information may also include one or more of the following (which may be referred to as model test related information): one or more model inference results of the first model during the operation of the second timer; actual results associated with one or more model inference results of the first model during the operation of the second timer; the number of inferences that meet the accuracy requirements in one or more model inference results of the first model during the operation of the second timer; the number of inferences that do not meet the accuracy requirements in one or more model inference results of the first model during the operation of the second timer; the ratio information or ratio information between the number of inferences that meet the accuracy requirements in one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model; the ratio information or ratio information between the number of inferences that do not meet the accuracy requirements in one or more model inference results of the first model during the operation of the second timer and the total number of inferences of the first model.

[0106] In some implementations, in order to allow the second device to know the distribution of the inference results of the first model so that the second device can comprehensively analyze the test performance of the first model, one or more model inference results of the first model during the operation of the above-mentioned second timer and the actual results corresponding to the one or more model inference results can be reported through the second information in chronological order.

[0107] As previously described, the second timer can be associated with multiple models. For example, the multiple models associated with the second timer include a first model and a second model. Assuming that after the second timer times out, the first timer associated with the second model has not timed out. In some implementations, the second information may further include an identifier of the second model, where the identifier of the second model indicates that the first timer associated with the second model has not timed out during the operation of the second timer.

[0108] It should be noted that even if the first timer associated with the second model does not time out during the operation of the second timer, this does not mean that the second model meets the accuracy requirements. Whether the second model meets the accuracy requirements can be determined based on the inference results of the second model during the operation of the second timer and the actual result information associated with the inference results.

[0109] In some implementations, the test-related information of the second model can be carried by the second information. That is, the multiple models associated with the second information can include the test-related information of the model whose first timer has expired and the test-related information of the model whose first timer has not expired. That is, the above-mentioned second information may include one or more of the following: the identification of one or more models participating in the model test during the operation of the second timer, one or more model inference results of each of the one or more first models during the operation of the second timer; the actual results associated with the one or more model inference results of each of the one or more first models during the operation of the second timer; the number of inferences that meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer; the number of inferences that do not meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer; the ratio information or ratio information between the number of inferences that meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer and the total number of model inferences of the corresponding first model; the ratio information or ratio information between the number of inferences that do not meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer and the total number of inferences of the corresponding first model.

[0110] In some implementations, if the second timer is associated with one or more first models, the method further includes: when the second timer times out, the first device stops running the one or more first models associated with the second timer.

[0111] Implementation method 3: Assume that the fourth parameter is used to configure the running time information of the first timer and the running time information of the second timer (ie, a combination of Example 1 and Example 2 above).

[0112] In some implementations, the method further includes: during the operation of the second timer, when the first timer times out, the first device recording an identifier of the first model associated with the first timer that times out and information about the number of timeouts of the first timer; and / or, when the second timer times out, the first device sending second information to the second device, and the second device correspondingly receiving the second information sent by the first device, the second information including an identifier of the first model associated with the first timer that times out at least once and information about the number of timeouts of the first timer. The identifier of the first model can be used to indicate that the first timer associated with the first model has timed out at least once.

[0113] In some implementations, the first device records the identifier of the first model associated with the first timer that times out and the timeout count information of the first timer, including: if the information entry associated with the first timer that times out is not recorded in the cache of the first device, the first device creates a new information entry, and records the identifier of the first model associated with the first timer that times out in the newly created information entry, and records the timeout count information of the first timer as 1, wherein the initial value of the timeout count information of the first timer can be 0; and / or, if the information entry associated with the first timer that times out is recorded in the cache of the first device, the first device adds 1 to the value of the stop count information of the first timer recorded in the corresponding information entry.

[0114] For example, the number of times the first timer times out can be counted by a counter associated with the first timer, and accordingly, the initial value of the counter can be 0. If the cache of the first device does not record an information entry associated with the timed-out first timer, the count of the counter associated with the first timer becomes 1; and / or if the cache of the first device records an information entry associated with the timed-out first timer, the first device increments the count of the counter associated with the first timer by 1.

[0115] In an embodiment of the present application, during the operation of the second timer, if the first model associated with the first timer times out, the first device does not immediately send the second information, nor does it need to immediately stop running the first model. It only needs to temporarily record the identifier corresponding to the first model and the number of timeouts of the first timer. Thereafter, the first model will continue to run, that is, the test of the first model will continue. Thereafter, when the first timer times out again, the number of timeouts of the first timer is updated, until the second timer times out, the first device sends the second information to the second device. The second information includes all information items recorded during the operation of the second timer, which helps to reduce the overhead required to transmit the second information. It also helps to avoid the situation where a first model is judged to be of good performance based on a single timeout of the first timer, and helps to improve the accuracy of the test of the first model.

[0116] For example, the second timer runs for 60 seconds, and during the second timer's execution, Models A, B, and C need to be tested. Assume that after the first device receives the first message, the second timer begins running. Ten seconds into the second timer's execution, the first timer associated with Model A times out. The first device can record Model A's identifier and the timeout count of the first timer associated with Model A, which is 1. Model A continues to run. Then, at the 16th second of the second timer's execution, the first timer associated with Model B times out. The first device can record Model B's identifier and the timeout count of the first timer associated with Model B, which is 1. Model B continues to run. Then, at the 20th second of the second timer's execution, the first timer associated with Model A times out. The first device can update the timeout count of the first timer associated with Model A to 2. Model A continues to run. At the 30th second of the second timer's execution, the first timer associated with Model B times out. The first device can update the timeout count of the first timer associated with Model B to 2. Model B continues to run. Then, at the 32nd second of the second timer's operation, the first timer associated with model A times out. At this time, the first device can update the timeout count of the first timer associated with model A to 3, and model A continues to run. Then, at the 50th second of the second timer's operation, the first timer associated with model C times out. At this time, the first device can record the identifier of model C and the timeout count of the first timer associated with model C, which is 1. At this time, model C continues to run. When the second timer times out (i.e., 60 seconds after the second timer's operation), the first device reports second information to the second device, where the second information includes the identifier of model A and the timeout count of the first timer associated with model A, which is 3; the identifier of model B and the timeout count of the first timer associated with model B, which is 2; and the identifier of model C and the timeout count of the first timer associated with model C, which is 1. In other words, the identifier of model A, the identifier of model B, the identifier of model C, and the timeout count of the first timer associated with each model can all be transmitted through the second information, which helps reduce the resources required to transmit the second information.

[0117] It should be noted that, based on the above introduction, the number of timeouts of the above-mentioned first timer can be understood as the number of timeouts of the first timer associated with each model when the second timer times out.

[0118] In the embodiment of the present application, the second information is not specifically limited. In some implementations, the second information may also include one or more of the following: one or more model inference results of the first model during the operation of the second timer; actual results associated with one or more model inference results of the first model during the operation of the second timer; the number of inferences that meet the accuracy requirements among one or more model inference results of the first model during the operation of the second timer; the number of inferences that do not meet the accuracy requirements among one or more model inference results of the first model during the operation of the second timer; the ratio information or ratio information between the number of inferences that meet the accuracy requirements among one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model; the ratio information or ratio information between the number of inferences that do not meet the accuracy requirements among one or more model inference results of the first model during the operation of the second timer and the total number of inferences of the first model.

[0119] In some implementations, in order to allow the second device to know the distribution of the inference results of the first model so that the second device can comprehensively analyze the test performance of the first model, one or more model inference results of the first model during the operation of the above-mentioned second timer and the actual results corresponding to the one or more model inference results can be reported through the second information in chronological order.

[0120] As previously described, the second timer can be associated with multiple models. For example, the multiple models associated with the second timer include a first model and a second model. Assuming that after the second timer times out, the first timer associated with the second model has not timed out. In some implementations, the second information may further include an identifier of the second model, where the identifier of the second model indicates that the first timer associated with the second model has not timed out during the operation of the second timer.

[0121] It should be noted that even if the first timer associated with the second model does not time out during the operation of the second timer, this does not mean that the second model meets the accuracy requirements. Whether the second model meets the accuracy requirements can be determined based on the inference results of the second model during the operation of the second timer and the actual result information associated with the inference results.

[0122] In some implementations, the test-related information of the second model can be carried by the second information. That is, the multiple models associated with the second information can include the test-related information of the model whose first timer has expired and the test-related information of the model whose first timer has not expired. That is, the above-mentioned second information may include one or more of the following: the identification of one or more models participating in the model test during the operation of the second timer, one or more model inference results of each of the one or more first models during the operation of the second timer; the actual results associated with the one or more model inference results of each of the one or more first models during the operation of the second timer; the number of inferences that meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer; the number of inferences that do not meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer; the ratio information or ratio information between the number of inferences that meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer and the total number of model inferences of the corresponding first model; the ratio information or ratio information between the number of inferences that do not meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer and the total number of inferences of the corresponding first model.

[0123] In some implementations, if the second timer is associated with one or more first models, the method further includes: when the second timer times out, the first device stops running the one or more first models associated with the second timer.

[0124] Implementation method 4: Assume that the fourth parameter is used to configure the running time information of the second timer (ie, Example 2 above).

[0125] In some implementations, the method further includes: when the second timer expires, the first device sends second information to the second device, and accordingly, the second device receives the second information sent by the first device, the second information including one or more model test results recorded during the operation of the second timer, and the models corresponding to the model test results recorded during the operation of the second timer (hereinafter referred to as the models tested during the operation of the second timer) include one or more first models. The model test results recorded during the operation of the second timer may include one or more inference results of the corresponding model.

[0126] In an embodiment of the present application, if the second timer times out, the first device may send a second message to the second device to indicate one or more first model test results recorded during the operation of the second timer, which helps to reduce the complexity of recording the test results.

[0127] In the embodiment of the present application, the second information is not specifically limited. In some implementations, the second information may further include one or more of the following: identifications of one or more models participating in the model test during the second timer operation; one or more model inference results of each of the one or more first models during the second timer operation; actual results associated with one or more model inference results of each of the one or more first models during the second timer operation; the number of inferences that meet the accuracy requirements among the one or more model inference results of each of the one or more first models during the second timer operation; the number of inferences that do not meet the accuracy requirements among the one or more model inference results of each of the one or more first models during the second timer operation; the ratio or value information between the number of inferences that meet the accuracy requirements among the one or more model inference results of each of the one or more first models during the second timer operation and the total number of model inferences of the corresponding first model; the ratio or value information between the number of inferences that do not meet the accuracy requirements among the one or more model inference results of each of the one or more first models during the second timer operation and the total number of inferences of the corresponding first model.

[0128] In some implementations, in order to allow the second device to know the distribution of the inference results of the first model so that the second device can comprehensively analyze the test performance of the first model, one or more model inference results in the test results of each first model during the operation of the above-mentioned second timer and the actual results corresponding to the one or more model inference results can be reported through the second information in chronological order.

[0129] In some implementations, if the second timer is associated with one or more first models, the method further includes: when the second timer times out, the first device stops running the one or more first models associated with the second timer.

[0130] Implementation method 4: Assume that the fourth parameter is used to configure the target inference number threshold (ie, combined with Example 3 above).

[0131] In some implementations, the method further includes: when the number of inferences for testing the first model reaches a target inference number threshold, the first device sending second information to the second device, and the second device receiving the second information sent by the first device, the second information including a test result of testing the first model. The test result of the first model may include one or more inference results of the first model.

[0132] In an embodiment of the present application, if the number of inferences tested on the first model reaches the target inference number threshold, the first device can send second information to the second device, that is, the test results of different models can be sent separately, which helps to avoid the first device using the cache to record the test results of each model for a long time, so as to reduce the occupancy of the cache space.

[0133] In the embodiments of the present application, the second information is not specifically limited. In some implementations, the second information may further include one or more of the following: an identifier of the first model; one or more model inference results of the first model; actual results associated with one or more model inference results of the first model; the number of inferences that meet the accuracy requirements among one or more model inference results of the first model; the number of inferences that do not meet the accuracy requirements among one or more model inference results of the first model; the ratio or value information between the number of inferences that meet the accuracy requirements among one or more model inference results of the first model and the total number of model inferences of the first model; the ratio or value information between the number of inferences that do not meet the accuracy requirements among one or more model inference results of the first model and the total number of inferences of the first model.

[0134] In some implementations, in order to allow the second device to know the distribution of the inference results of the first model so that the second device can comprehensively analyze the test performance of the first model, one or more model inference results in the test results of the above-mentioned first model and the actual results corresponding to the one or more model inference results can be reported through the second information in chronological order.

[0135] Implementation method 5: Assume that the fourth parameter is used to configure the target inference number threshold and the running time information of the second timer (ie, a combination of Example 3 and Example 2 above).

[0136] In some implementations, the above method also includes: during the operation of the second timer, when the number of inferences tested on the first model reaches the target inference number threshold, the first device stops running the first model whose inference number reaches the target inference number threshold; and / or, when the second timer times out, the first device sends second information to the second device, and accordingly, the second device receives the second information sent by the first device, the second information includes the model test results recorded during the operation of the second timer, and the model corresponding to the model test results recorded during the operation of the second timer includes the first model.

[0137] It should be noted that the model test results recorded during the second timer operation may include one or more inference results of the corresponding model. For example, if the model test results recorded during the second timer operation include the test results of the first model, the test results of the first model may include one or more inference results of the first model.

[0138] In an embodiment of the present application, if the number of inferences tested on the first model reaches a target inference threshold, the first device may first record the identifier of the first model, stop running the first model, and then send the second information to the second device after the second timer expires. In other words, the first device does not need to immediately send the second information when the number of inferences of the first model reaches the target inference threshold. Instead, it only needs to temporarily record the identifier of the first model and wait until the second timer expires before sending it together with the second information, which helps to save the overhead required to transmit the second information.

[0139] For example, the running time of the second timer is 60 seconds, and model A and model B need to be tested during the running of the second timer. Assume that when the first device receives the first information, the second timer starts running, and 30 seconds after the second timer runs, the number of inferences of model A reaches the target inference number threshold. At this time, the first device can stop running model A and record the identifier of model A. Then, at the 50th second after the second timer runs, the number of inferences of model B reaches the target inference number threshold. At this time, the first device can stop running model B and record the identifier of model B. When the second timer times out (that is, 60 seconds after the second timer runs), the first device reports the second information to the second device, where the second information includes the identifier of model A and the identifier of model B. In other words, the identifier of model A and the identifier of model B can both be transmitted through the second information, which helps to reduce the resources required to transmit the second information.

[0140] During the operation of the second timer, the first device may record the identifier of the first model whose number of inferences reaches the target inference number threshold for subsequent transmission via the second information. For example, if the number of inferences tested on the first model reaches the target inference number threshold, the first device may record the identifier of the first model as a new information entry in the cache.

[0141] In the embodiment of the present application, the second information is not specifically limited. In some implementations, the second information may also include one or more of the following: an identifier of the first model, one or more model inference results of the first model during the operation of the second timer; an actual result associated with one or more model inference results of the first model during the operation of the second timer; the number of inferences that meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer; the number of inferences that do not meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer; the ratio information or ratio information between the number of inferences that meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model; the ratio information or ratio information between the number of inferences that do not meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer and the total number of inferences of the first model.

[0142] In some implementations, in order to allow the second device to know the distribution of the inference results of the first model so that the second device can comprehensively analyze the test performance of the first model, the test results of the first model during the operation of the above-mentioned second timer include one or more model inference results and the actual results corresponding to one or more model inference results can be reported through the second information in chronological order.

[0143] As previously described, the second timer can be associated with multiple models. Accordingly, the second information can include the identifiers of the models that actually participated in the model test during the operation of the second timer. Taking the multiple models associated with the second timer including the above-mentioned first model and the second model as an example, assuming that after the second timer times out, the number of inferences of the second model does not reach the target inference number threshold. In some implementations, the above-mentioned second information can also carry the identifier of the second model, wherein the identifier of the second model indicates that the number of inferences of the second model has not reached the target inference number threshold.

[0144] It should be noted that although the number of inferences of the second model does not reach the target inference number threshold, it does not mean that the test results of the second model do not meet the accuracy requirements. Whether the test results of the second model meet the accuracy requirements can be determined based on the model inference results of the second model during the operation of the second timer and the actual result information corresponding to the model inference results.

[0145] In some implementations, information related to the inference results of the second model may be carried by the second information. That is, the multiple models associated with the second information may include information related to the inference results of models whose inference times reach a target inference times threshold and information related to the inference results of models whose inference times do not reach the target inference times threshold. That is, the above-mentioned second information may include one or more of the following: the identification of one or more models participating in the model test during the operation of the second timer, one or more model inference results of each of the one or more first models during the operation of the second timer; the actual results associated with the one or more model inference results of each of the one or more first models during the operation of the second timer; the number of inferences that meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer; the number of inferences that do not meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer; the ratio information or ratio information between the number of inferences that meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer and the total number of model inferences of the corresponding first model; the ratio information or ratio information between the number of inferences that do not meet the accuracy requirements in the one or more model inference results of each of the one or more first models during the operation of the second timer and the total number of inferences of the corresponding first model.

[0146] In some implementations, if the second timer is associated with one or more first models, the method further includes: when the second timer times out, the first device stops running the one or more first models associated with the second timer.

[0147] In some implementations, the above method also includes: when the first condition is met, the first device performs a first operation, and the first operation includes one or more of the following operations: stopping the running of one or more first models being tested in the first device; stopping the timer related to the first model test running in the first device; deleting the configuration contained in the stored first information; clearing the record information associated with the test process of the first model recorded in the cache of the first device.

[0148] If the first operation includes stopping a timer related to the first model test that is running in the first device, the timer related to the first model test may include, for example, the first timer and / or the second timer described above.

[0149] If the first operation includes deleting the configuration contained in the stored first information, the configuration contained in the first information may be, for example, one or more of the identifier of the first model mentioned above; the first parameter; the second parameter; the third parameter and the fourth parameter.

[0150] If the first operation includes clearing the record information associated with the test process of the first model recorded in the cache of the first device, the record information associated with the test process of the first model can be, for example, one or more of the information stored in the cache introduced above, or the record information associated with the test process of the first model can be, for example, one or more of the information prepared to be carried in the second information introduced above.

[0151] In some implementations, the first condition includes one or more of the following: the first device triggers connection re-establishment; the first device triggers wireless link failure; the first device triggers beam failure; the first device receives synchronization reconfiguration information; the first device performs switching; the first device receives a connection release message; the first device receives a first instruction sent by the second device, and the first instruction is used to instruct the first device to stop testing the first model.

[0152] In some implementations, the method further includes the second device sending a first instruction to the first device, where the first instruction is used to instruct the first device to stop testing the first model.

[0153] In some implementations, the above method also includes: the first device sends a first suggestion message to the second device, and correspondingly, the second device receives the first suggestion message sent by the first device, the first suggestion message includes the model identifier of the model that the first device expects to suspend testing, or the first suggestion message is used to indicate to the second device that the first device expects to suspend all model tests.

[0154] In some implementations, the above method also includes: the first device receives the first indication information sent by the second device, and correspondingly, the second device sends the first indication information to the first device, and the first indication information is used to indicate whether the first device is allowed to send the first suggestion information. At this time, if the first indication information indicates that the first device is allowed to send the first suggestion information, the first device can send the first suggestion information to the second device. On the contrary, if the first indication information indicates that the first device is not allowed to send the first suggestion information, the first device cannot send the first suggestion information to the second device. Of course, in an embodiment of the present application, the first suggestion information can also be sent directly by the first device without first determining whether it can be sent based on the first indication information.

[0155] In some implementations, the above method also includes: the first device sends first capability information to the second device, and correspondingly, the second device receives the first capability information sent by the first device, and the first capability information is used to indicate one or more of the following: whether the first device supports the model testing function; one or more model identifiers in the first device that support the model testing function; one or more model identifiers in the first device that do not support the model testing function; the number of models supported by the first device for simultaneous model testing; whether the first device supports reporting first suggestion information, the first suggestion information indicates the model that the first device expects to suspend testing, or the first suggestion information is used to indicate that the first device expects to suspend all model tests.

[0156] In an embodiment of the present application, any information involved above (for example, one or more of the first information, the second information, the first suggestion information, the first indication information, and the first capability information) can be carried by one or more of the following message types: NR Positioning Protocol A (NRPPa) message, long term evolution positioning protocol (LPP) message, non-access stratum (NAS) message, radio resource control (RRC) message, media access control control element (MAC CE), downlink control information (DCI), uplink control information (UCI), physical uplink control channel (PUCCH), physical uplink shared channel (PUSCH), inter-node message, Xn interface message, F1 interface message, E1 interface message, NG interface message, core network service architecture-based message or AI dedicated message.

[0157] In an embodiment of the present application, any of the information mentioned above can be carried by one or more of unicast messages, multicast messages and broadcast messages.

[0158] Unicast messages can be understood as one-to-one transmissions, meaning they are sent from one sender to one receiver. The source delivers unicast messages over a unicast channel, and only devices or network equipment allocated the corresponding unicast resources can attempt to receive them. Unicast messages are also called dedicated signaling.

[0159] The multicast message described above can be understood as a one-to-many transmission, meaning it's sent from a single sender to multiple receivers. The source transmits the message via a multicast channel. Terminal devices or network devices within the multicast signal's coverage area and that are members of the group can attempt to receive the message. When a terminal device or network device joins a group, it acquires multicast channel resources.

[0160] The broadcast message can be understood as a message transmitted from one sender to any receiver. The source transmits the message via a broadcast channel, and any terminal or network device within the coverage area of ​​the broadcast signal can attempt to receive it.

[0161] In the embodiments of the present application, the first device and / or the second device are not limited. For example, the first device is a terminal device, and the second device is a network device. In another example, the first device is a network device, and the second device is a terminal device. In another example, the first device is a first terminal device, and the second device is a second terminal device. In another example, the first device is a first network device, and the second device is a second network device.

[0162] The network device is an access network device, a core network device, an AI / ML model-related information management device, or an operation administration and maintenance (OAM) device. Exemplarily, the access network device is any one of the following: a gNB, a centralized unit (CU), a distributed unit (DU), a centralized unit-control plane (CU-CP), or a centralized unit-user plane (CU-UP).

[0163] Exemplarily, the core network device is any one of the following: location management function (LMF) network element, network slice selection function (NSSF), authentication server function (AUSF), unified data management (UDM), access and mobility management function (AMF), session management function (SMF), policy control function (PCF), user plane function (UPF), sensing control function (SF), network data analysis (NWDAF) network element.

[0164] The method embodiment of the present application is described in detail above in conjunction with Figures 1 and 2. The device embodiment of the present application is described in detail below in conjunction with Figures 3 to 5. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment. Therefore, for parts not described in detail, reference can be made to the above method embodiment.

[0165] FIG3 is a schematic diagram of a communication device according to an embodiment of the present application. The communication device 300 shown in FIG3 is a first device, and the communication device 300 includes a receiving unit 310 .

[0166] The receiving unit 310 is configured to receive first information sent by a second device, where the first information is used to perform a performance test and / or a performance evaluation on a first model that is not activated and used in the first device.

[0167] In one possible implementation, the first information includes one or more of the following: an identifier of the first model; a first parameter, the first parameter is used to determine the activation time of the first model; a second parameter, the second parameter is used to determine whether the inference result of the first model meets the accuracy requirements; a third parameter, the third parameter is used to configure a reference signal for testing the first model; and a fourth parameter, the fourth parameter is associated with the test process of the first model.

[0168] In one possible implementation, if the first information includes the first parameter, the first parameter is used to configure the first delay information, and the first model is activated after the time length corresponding to the first delay information has elapsed from the moment the first device receives the first information; or the first parameter is used to configure the absolute time information for activating the first model.

[0169] In one possible implementation, if the first information includes the second parameter, and the second parameter includes an accuracy threshold, if the absolute value of the difference between the inference result of the first model and the actual result is less than or equal to the accuracy threshold, then the inference result of the first model is considered to meet the accuracy requirements.

[0170] In a possible implementation, if the first information includes the fourth parameter, the fourth parameter is used to configure the running time information of the first timer, and the first timer is associated with the inference result of the first model.

[0171] In one possible implementation, the communication device also includes: when the first timer times out, a first sending unit is used to send second information to the second device, the second information includes an identifier of the first model associated with the first timer that times out, and the identifier of the first model associated with the first timer indicates that the first timer associated with the corresponding model has timed out.

[0172] In a possible implementation, the communication device further includes: when the first timer times out, a first processing unit configured to stop running a first model associated with the first timer that has timed out.

[0173] In a possible implementation, the fourth parameter is used to configure running time information of a second timer, and the second timer is associated with a test result reporting operation of the first model.

[0174] In one possible implementation, the communication device also includes: during the operation of the second timer, when the first timer times out, a second processing unit, used to record the identifier of the first model associated with the first timer that times out, and stop running the first model associated with the first timer that times out; and / or, when the second timer times out, a second sending unit, used to send second information to the second device, the second information including the identifier of the first model associated with the first timer that times out, the identifier of the first model associated with the first timer indicating that the first timer associated with the corresponding model has timed out.

[0175] In one possible implementation, the communication device also includes: during the operation of the second timer, when the first timer times out, a third processing unit, used to record the identifier of the first model associated with the first timer that times out and the number of timeouts of the first timer; and / or, when the second timer times out, a third sending unit, used to send second information to the second device, the second information including the identifier of the first model associated with the first timer that times out at least once and the number of timeouts of the first timer.

[0176] In one possible implementation, if the information entry associated with the first timer that has timed out is not recorded in the cache of the first device, the third processing unit is used to create a new information entry, and record the identifier of the first model associated with the first timer that has timed out in the new information entry, and record the timeout number information of the first timer as 1; and / or, if the information entry associated with the first timer that has timed out is recorded in the cache of the first device, the third processing unit is used to add 1 to the value of the stop number information of the first timer recorded in the corresponding information entry.

[0177] In one possible implementation, a start condition of the first timer includes that the inference result of the first model does not meet the accuracy requirement; and / or a stop condition of the first timer includes that the inference result of the first model meets the accuracy requirement.

[0178] In one possible implementation, the start condition of the first timer includes that the reasoning results of the first model for M consecutive times do not meet the accuracy requirements, where M is a positive integer greater than or equal to 1; and / or the stop condition of the first timer includes that the reasoning results of the first model for N consecutive times meet the accuracy requirements, where N is a positive integer greater than or equal to 1.

[0179] In a possible implementation, the fourth parameter is used to configure running time information of a second timer, and the second timer is associated with a test result reporting operation of the first model.

[0180] In one possible implementation, the communication device also includes: when the second timer times out, a fourth sending unit, used to send second information to the second device, the second information including the model test results recorded during the operation of the second timer, and the model corresponding to the model test results recorded during the operation of the second timer includes the first model.

[0181] In a possible implementation, if the first information includes the fourth parameter, the fourth parameter is used to configure a target inference times threshold for testing the first model.

[0182] In one possible implementation, the communication device further includes: when the number of inferences for testing the first model reaches the target inference number threshold, a fifth sending unit is used to send second information to the second device, where the second information includes a test result of testing the first model.

[0183] In a possible implementation, the fourth parameter is used to configure the duration of a second timer, and the second timer is associated with a test result reporting operation of the first model.

[0184] In one possible implementation, the communication device further includes: during the operation of the second timer, when the number of inferences tested on the first model reaches the target inference number threshold, a fourth processing unit, used to stop running the first model whose number of inferences reaches the target inference number threshold; and / or, when the second timer times out, a sixth sending unit, used to send second information to the second device, the second information including the model test results recorded during the operation of the second timer, and the models corresponding to the model test results recorded during the operation of the second timer include the first model.

[0185] In one possible implementation, the second information includes one or more of the following: one or more model inference results of the first model during the operation of the second timer; actual results associated with one or more model inference results of the first model during the operation of the second timer; the number of inferences that meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer; the number of inferences that do not meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer; the ratio information or ratio information between the number of inferences that meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer; the ratio information or ratio information between the number of inferences that do not meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer.

[0186] In a possible implementation, the second timer is associated with one or more first models, and the communication device further includes: when the second timer times out, a fifth processing unit, configured to stop running the one or more first models associated with the second timer.

[0187] In a possible implementation, the start time of the second timer is determined based on the time when the first device receives the first information, or the start time of the second timer is determined based on the time when the first device activates the first model.

[0188] In one possible implementation, the communication device also includes: when the first condition is met, a sixth processing unit, used to perform a first operation, the first operation including one or more of the following operations: stopping the running of one or more first models being tested in the first device; stopping the timer related to the first model test running in the first device; deleting the configuration contained in the stored first information; clearing the record information associated with the test process of the first model recorded in the cache of the first device.

[0189] In one possible implementation, the first condition includes one or more of the following: the first device triggers connection re-establishment; the first device triggers wireless link failure; the first device triggers beam failure; the first device receives synchronization reconfiguration information; the first device performs switching; the first device receives a connection release message; the first device receives a first instruction sent by the second device, and the first instruction instructs the first device to stop testing the first model.

[0190] In one possible implementation, the communication device also includes: a seventh sending unit, used to send a first suggestion message to the second device, the first suggestion message including a model identifier of the first device that expects to suspend testing or the first suggestion message is used to indicate to the second device that the first device expects to suspend all model tests.

[0191] In one possible implementation, the communication device also includes: an eighth sending unit, used to send first capability information to the second device, the first capability information being used to indicate one or more of the following: whether the first device supports the model testing function; one or more model identifiers in the first device that support the model testing function; one or more model identifiers in the first device that do not support the model testing function; the number of models supported by the first device for simultaneous model testing; whether the first device supports reporting first suggestion information, the first suggestion information indicating the model that the first device expects to suspend testing, or the first suggestion information being used to indicate that the first device expects to suspend all model tests.

[0192] FIG4 is a schematic diagram of a communication device according to an embodiment of the present application. The communication device 400 shown in FIG4 is a second device, and the communication device 400 includes a sending unit 410 .

[0193] The sending unit 410 is configured to send first information to a first device, where the first information is used to perform a performance test and / or a performance evaluation on a first model that is not activated for use by the first device.

[0194] In one possible implementation, the first information includes one or more of the following: an identifier of the first model; a first parameter, the first parameter is used to determine the activation time of the first model; a second parameter, the second parameter is used to determine whether the inference result of the first model meets the accuracy requirements; a third parameter, the third parameter is used to configure a reference signal for testing the first model; and a fourth parameter, the fourth parameter is associated with the inference result of the first model.

[0195] In one possible implementation, if the first information includes the first parameter, the first parameter is used to configure the first delay information, and the first model is activated after the time length corresponding to the first delay information has elapsed from the moment the first device receives the first information; or the first parameter is used to configure the absolute time information for activating the first model.

[0196] In one possible implementation, if the first information includes the second parameter, and the second parameter includes an accuracy threshold, if the absolute value of the difference between the inference result of the first model and the actual result is less than or equal to the accuracy threshold, then the inference result of the first model is considered to meet the accuracy requirements.

[0197] In a possible implementation, if the first information includes the fourth parameter, the fourth parameter is used to configure the running time information of the first timer, and the first timer is associated with the inference result of the first model.

[0198] In one possible implementation, the communication device also includes: a first receiving unit for receiving second information sent by the first device, the second information including an identifier of a first model associated with a first timer that has timed out, the identifier of the first model associated with the first timer indicating that a first timer associated with the corresponding model has timed out.

[0199] In a possible implementation, the fourth parameter is used to configure running time information of a second timer, and the second timer is associated with a test result reporting operation of the first model.

[0200] In one possible implementation, the communication device also includes: a second receiving unit for receiving second information sent by the first device, the second information including an identifier of a first model associated with a first timer that has timed out, the identifier of the first model associated with the first timer indicating that a timeout has occurred in the first timer associated with the corresponding model, wherein the information contained in the second information is recorded during the operation of the second timer.

[0201] In one possible implementation, the communication device also includes: a third receiving unit, used to receive second information sent by the first device, the second information including an identifier of a first model associated with a first timer that has timed out at least once and information about the number of timeouts of the first timer, wherein the information contained in the second information is recorded during the operation of the second timer.

[0202] In a possible implementation, the fourth parameter is used to configure running time information of a second timer, and the second timer is associated with a test result reporting operation of the first model.

[0203] In one possible implementation, the communication device also includes: a fourth receiving unit, used to receive second information sent by the first device, the second information includes a model test result recorded during the operation of the second timer, and the model corresponding to the model test result recorded during the operation of the second timer includes the first model.

[0204] In a possible implementation, if the first information includes the fourth parameter, the fourth parameter is used to configure a target inference times threshold for testing the first model.

[0205] In a possible implementation, the communication device further includes: a fifth receiving unit, configured to receive second information sent by the first device, where the second information includes a test result of testing the first model.

[0206] In a possible implementation, the fourth parameter is used to configure the duration of a second timer, and the second timer is associated with a test result reporting operation of the first model.

[0207] In one possible implementation, the communication device also includes: a sixth receiving unit, used to receive second information sent by the first device, the second information includes a model test result recorded during the operation of the second timer, and the model corresponding to the model test result recorded during the operation of the second timer includes the first model.

[0208] In one possible implementation, the second information includes one or more of the following: one or more model inference results of the first model during the operation of the second timer; actual results associated with one or more model inference results of the first model during the operation of the second timer; the number of inferences that meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer; the number of inferences that do not meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer; the ratio information or ratio information between the number of inferences that meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer; the ratio information or ratio information between the number of inferences that do not meet the accuracy requirements among the one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer.

[0209] In one possible implementation, the communication device also includes: a seventh receiving unit, used to receive a first suggestion message sent by the first device, the first suggestion message including a model identifier of the model that the first device expects to suspend testing, or the first suggestion message is used to indicate to the second device that the first device expects to suspend all model tests.

[0210] In one possible implementation, the communication device also includes: an eighth receiving unit, used to receive first capability information sent by the first device, the first capability information is used to indicate one or more of the following: whether the first device supports the model testing function; one or more model identifiers in the first device that support the model testing function; one or more model identifiers in the first device that do not support the model testing function; the number of models supported by the first device for simultaneous model testing; whether the first device supports reporting first suggestion information, the first suggestion information indicates the model that the first device expects to suspend testing, or the first suggestion information is used to indicate that the first device expects to suspend all model tests.

[0211] In an optional embodiment, the receiving unit 310 may be a transceiver 530. The communication device 300 may further include a processor 510 and a memory 520, as specifically shown in FIG5 .

[0212] In an optional embodiment, the sending unit 410 may be a transceiver 530. The communication device 400 may further include a processor 510 and a memory 520, as specifically shown in FIG5 .

[0213] Figure 5 is a schematic block diagram of a communication device according to an embodiment of the present application. The dashed lines in Figure 5 indicate that the unit or module is optional. The device 500 may be used to implement the method described in the above method embodiment. The device 500 may be a chip, a terminal device, or a network device.

[0214] The device 500 may include one or more processors 510. The processor 510 may support the device 500 to implement the method described in the method embodiment above. The processor 510 may be a general-purpose processor or a special-purpose processor. For example, the processor may be a central processing unit (CPU). Alternatively, the processor may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0215] The apparatus 500 may further include one or more memories 520. The memories 520 store programs that can be executed by the processor 510, causing the processor 510 to perform the methods described in the above method embodiments. The memories 520 may be independent of the processor 510 or integrated into the processor 510.

[0216] The apparatus 500 may further include a transceiver 530. The processor 510 may communicate with other devices or chips via the transceiver 530. For example, the processor 510 may transmit and receive data with other devices or chips via the transceiver 530.

[0217] The present application also provides a computer-readable storage medium for storing a program. The computer-readable storage medium can be applied to a terminal or network device provided in the present application, and the program enables a computer to execute the method performed by the terminal or network device in each embodiment of the present application.

[0218] The present application also provides a computer program product. The computer program product includes a program. The computer program product can be applied to a terminal or network device provided in the present application, and the program causes a computer to execute the method performed by the terminal or network device in each embodiment of the present application.

[0219] The embodiments of the present application also provide a computer program. The computer program can be applied to the terminal or network device provided in the embodiments of the present application, and the computer program enables a computer to execute the method performed by the terminal or network device in each embodiment of the present application.

[0220] It should be understood that the terms "system" and "network" in this application can be used interchangeably. In addition, the terms used in this application are only used to explain the specific embodiments of this application and are not intended to limit this application. The terms "first", "second", "third", and "fourth" in the specification and claims of this application and the accompanying drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0221] In the embodiments of this application, the term "indication" may refer to a direct indication, an indirect indication, or an indication of an association. For example, "A indicates B" may refer to a direct indication of B, e.g., B can obtain information through A; it may refer to an indirect indication of B, e.g., A indicates C, e.g., B can obtain information through C; or it may refer to an association between A and B.

[0222] In the embodiment of the present application, "B corresponding to A" means that B is associated with A and B can be determined based on A. However, it should be understood that determining B based on A does not mean determining B based solely on A, but B can also be determined based on A and / or other information.

[0223] In the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between the two, or an association relationship between the two, or a relationship between indication and indication, configuration and configuration, etc.

[0224] In the embodiments of the present application, "pre-definition" or "pre-configuration" may be implemented by pre-storing corresponding codes, tables, or other methods that can be used to indicate relevant information in a device (e.g., a terminal device and a network device). The present application does not limit the specific implementation method. For example, pre-definition may refer to information defined in a protocol.

[0225] In the embodiments of the present application, the “protocol” may refer to a standard protocol in the communications field, for example, it may include an LTE protocol, an NR protocol, and related protocols used in future communication systems, and the present application does not limit this.

[0226] In the embodiments of this application, the term "and / or" is simply a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0227] In various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0228] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0229] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0230] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0231] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be read by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0232] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A wireless communication method, characterized in that: include: The first device receives first information sent by the second device, where the first information is used to perform performance testing and / or performance evaluation on a first model that is not activated and used in the first device.

2. The method according to claim 1, characterized in that The first information includes one or more of the following: an identifier of the first model; A first parameter, wherein the first parameter is used to determine an activation time of the first model; a second parameter, wherein the second parameter is used to determine whether the inference result of the first model meets the accuracy requirement; a third parameter, the third parameter being used to configure a reference signal for testing the first model; A fourth parameter is associated with the testing process of the first model.

3. The method according to claim 2, characterized in that If the first information includes the first parameter, the first parameter is used to configure the first delay information, and the first device activates the first model after a time period corresponding to the first delay information has elapsed since the first device received the first information; or, The first parameter is used to configure the absolute time information for activating the first model.

4. The method according to claim 2 or 3, characterized in that If the first information includes the second parameter, and the second parameter includes an accuracy threshold, if the absolute value of the difference between the inference result of the first model and the actual result is less than or equal to the accuracy threshold, it is considered that the inference result of the first model meets the accuracy requirements.

5. The method according to any one of claims 2 to 4, characterized in that: If the first information includes the fourth parameter, the fourth parameter is used to configure the running time information of the first timer, and the first timer is associated with the inference result of the first model.

6. The method according to claim 5, characterized in that The method further comprises: When the first timer times out, the first device sends second information to the second device, wherein the second information includes an identifier of a first model associated with the first timer that times out, and the identifier of the first model associated with the first timer indicates that the first timer associated with the corresponding model times out.

7. The method according to claim 5 or 6, characterized in that The method further comprises: When the first timer times out, the first device stops running the first model associated with the first timer that times out.

8. The method according to claim 5, characterized in that The fourth parameter is used to configure the running time information of the second timer, and the second timer is associated with the test result reporting operation of the first model.

9. The method according to claim 8, characterized in that The method further comprises: During the operation of the second timer, when the first timer times out, the first device records the identifier of the first model associated with the first timer that times out, and stops running the first model associated with the first timer that times out; and / or, When the second timer times out, the first device sends second information to the second device, wherein the second information includes an identifier of a first model associated with the first timer that times out, and the identifier of the first model associated with the first timer indicates that the first timer associated with the corresponding model times out.

10. The method according to claim 8, characterized in that The method further comprises: During the operation of the second timer, when the first timer times out, the first device records the identifier of the first model associated with the first timer that times out and the timeout number information of the first timer; and / or, When the second timer times out, the first device sends second information to the second device, where the second information includes an identifier of a first model associated with the first timer that has timed out at least once and timeout count information of the first timer.

11. The method according to claim 10, characterized in that The first device records an identifier of a first model associated with a first timer that times out and timeout information of the first timer, including: If the information entry associated with the first timer that has timed out is not recorded in the cache of the first device, the first device creates a new information entry, and records the identifier of the first model associated with the first timer that has timed out in the new information entry, and records the timeout number information of the first timer as 1; and / or, If an information entry associated with a first timer that has timed out is recorded in the cache of the first device, the first device adds 1 to the value of the number of stop times of the first timer recorded in the corresponding information entry.

12. The method according to any one of claims 5 to 11, characterized in that The start condition of the first timer includes that the inference result of the first model does not meet the accuracy requirement; and / or, The stop condition of the first timer includes that the inference result of the first model meets the accuracy requirement.

13. The method according to any one of claims 5 to 11, characterized in that: The start condition of the first timer includes that M consecutive inference results of the first model do not meet the accuracy requirement, where M is a positive integer greater than or equal to 1; and / or The stopping condition of the first timer includes that the N consecutive inference results of the first model meet the accuracy requirement, where N is a large A positive integer greater than or equal to 1.

14. The method according to any one of claims 2 to 4, characterized in that: The fourth parameter is used to configure the running time information of the second timer, and the second timer is associated with the test result reporting operation of the first model.

15. The method according to claim 14, characterized in that The method further comprises: When the second timer times out, the first device sends second information to the second device, where the second information includes the model test results recorded during the operation of the second timer, and the models corresponding to the model test results recorded during the operation of the second timer include the first model.

16. The method according to any one of claims 2 to 4, characterized in that: If the first information includes the fourth parameter, the fourth parameter is used to configure a target inference times threshold for testing the first model.

17. The method according to claim 16, characterized in that The method further comprises: When the number of inferences performed on the first model reaches the target inference number threshold, the first device sends second information to the second device, where the second information includes a test result of the first model.

18. The method according to claim 16, characterized in that The fourth parameter is used to configure the duration of a second timer, and the second timer is associated with the test result reporting operation of the first model.

19. The method according to claim 18, characterized in that The method further comprises: During the operation of the second timer, when the number of inferences for testing the first model reaches the target inference number threshold, the first device stops running the first model whose number of inferences reaches the target inference number threshold; and / or, When the second timer times out, the first device sends second information to the second device, where the second information includes the model test results recorded during the operation of the second timer, and the models corresponding to the model test results recorded during the operation of the second timer include the first model.

20. The method according to any one of claims 9 to 11, 15 and 19, characterized in that: The second information includes one or more of the following: one or more model inference results of the first model during the operation of the second timer; an actual result associated with one or more model inference results of the first model during the operation of the second timer; the number of inferences that meet the accuracy requirement among one or more model inference results of the first model during the operation of the second timer; The number of inferences that do not meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer; Proportional information or ratio information between the number of inferences that meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer; The ratio information or value information between the number of inferences that do not meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer.

21. The method according to any one of claims 8-11, 14-15 and 18-20, characterized in that The second timer is associated with one or more of the first models, and the method further includes: When the second timer times out, the first device stops running one or more first models associated with the second timer.

22. Any one of claims 8-11, 14-15 and 18-21, characterized in that The start time of the second timer is determined based on the time when the first device receives the first information, or the start time of the second timer is determined based on the time when the first device activates the first model.

23. The method according to any one of claims 1 to 22, characterized in that The method further comprises: When the first condition is met, the first device performs a first operation, where the first operation includes one or more of the following operations: Stop running one or more of the first models being tested in the first device; Stopping a timer associated with the first model test that is running in the first device; Deleting the configuration included in the stored first information; Clear the record information associated with the test process of the first model recorded in the cache of the first device.

24. The method of claim 23, wherein: The first condition includes one or more of the following: The first device triggers connection re-establishment; The first device triggers a radio link failure; The first device triggers a beam failure; The first device receives synchronization reconfiguration information; The first device performs switching; The first device receives a connection release message; The first device receives a first instruction sent by the second device, and the first instruction instructs the first device to stop the first model Conduct a test.

25. The method according to any one of claims 1 to 24, characterized in that The method further comprises: The first device sends first suggestion information to the second device, where the first suggestion information includes a model identifier of a model that the first device expects to suspend testing, or the first suggestion information is used to indicate to the second device that the first device expects to suspend all model tests.

26. The method according to any one of claims 1 to 25, characterized in that The method further comprises: The first device sends first capability information to the second device, where the first capability information is used to indicate one or more of the following: whether the first device supports a model testing function; One or more model identifiers supporting a model testing function in the first device; One or more model identifiers in the first device that do not support a model test function; The number of models supported by the first device for simultaneous model testing; Whether the first device supports reporting first suggestion information, where the first suggestion information indicates a model that the first device expects to suspend testing, or the first suggestion information is used to indicate that the first device expects to suspend all model tests.

27. A method of wireless communication, characterized in that: include: The second device sends first information to the first device, where the first information is used to perform performance testing and / or performance evaluation on a first model that is not activated for use by the first device.

28. The method of claim 27, wherein: The first information includes one or more of the following: an identifier of the first model; A first parameter, wherein the first parameter is used to determine an activation time of the first model; a second parameter, wherein the second parameter is used to determine whether the inference result of the first model meets the accuracy requirement; a third parameter, the third parameter being used to configure a reference signal for testing the first model; A fourth parameter, wherein the fourth parameter is associated with the inference result of the first model.

29. The method of claim 28, wherein: If the first information includes the first parameter, the first parameter is used to configure the first delay information, and the first model is activated after the first device receives the first information and the time corresponding to the first delay information has elapsed; or The first parameter is used to configure the absolute time information for activating the first model.

30. The method according to claim 28 or 29, characterized in that If the first information includes the second parameter, and the second parameter includes an accuracy threshold, if the absolute value of the difference between the inference result of the first model and the actual result is less than or equal to the accuracy threshold, it is considered that the inference result of the first model meets the accuracy requirements.

31. The method according to any one of claims 28 to 30, characterized in that If the first information includes the fourth parameter, the fourth parameter is used to configure the running time information of the first timer, and the first timer is associated with the inference result of the first model.

32. The method of claim 31, wherein: The method further comprises: The second device receives second information sent by the first device, where the second information includes an identifier of a first model associated with a first timer that has timed out, and the identifier of the first model associated with the first timer indicates that a first timer associated with the corresponding model has timed out.

33. The method of claim 31, wherein: The fourth parameter is used to configure the running time information of the second timer, and the second timer is associated with the test result reporting operation of the first model.

34. The method of claim 33, wherein: The method further comprises: The second device receives second information sent by the first device, the second information including an identifier of a first model associated with a first timer that has timed out, the identifier of the first model associated with the first timer indicating that a first timer associated with the corresponding model has timed out, wherein the second information includes information recorded during the operation of the second timer.

35. The method of claim 33, wherein: The method further comprises: The second device receives second information sent by the first device, wherein the second information includes an identifier of a first model associated with a first timer that has timed out at least once and timeout information of the first timer, wherein the information included in the second information is recorded during the operation of the second timer.

36. The method according to any one of claims 28 to 30, characterized in that The fourth parameter is used to configure the running time information of the second timer, and the second timer is associated with the test result reporting operation of the first model.

37. The method of claim 36, wherein: The method further comprises: The second device receives second information sent by the first device, where the second information includes model test results recorded during the operation of the second timer, and the models corresponding to the model test results recorded during the operation of the second timer include the first model.

38. The method according to any one of claims 28 to 30, characterized in that If the first information includes the fourth parameter, the fourth parameter is used to configure a target inference times threshold for testing the first model.

39. The method of claim 38, wherein: The method further comprises: The second device receives second information sent by the first device, where the second information includes a test result of testing the first model.

40. The method of claim 38, wherein: The fourth parameter is used to configure the duration of a second timer, and the second timer is associated with the test result reporting operation of the first model.

41. The method of claim 40, wherein: The method further comprises: The second device receives second information sent by the first device, where the second information includes model test results recorded during the operation of the second timer, and the models corresponding to the model test results recorded during the operation of the second timer include the first model.

42. The method according to any one of claims 34 to 35, 37 and 41, characterized in that The second information includes one or more of the following: one or more model inference results of the first model during the operation of the second timer; an actual result associated with one or more model inference results of the first model during the operation of the second timer; the number of inferences that meet the accuracy requirement among one or more model inference results of the first model during the operation of the second timer; The number of inferences that do not meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer; Proportional information or ratio information between the number of inferences that meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer; The ratio information or value information between the number of inferences that do not meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer.

43. The method according to any one of claims 27 to 42, characterized in that The method further comprises: The second device receives first suggestion information sent by the first device, where the first suggestion information includes a model identifier of a model that the first device expects to suspend testing, or the first suggestion information is used to indicate to the second device that the first device expects to suspend all model tests.

44. The method according to any one of claims 27 to 43, characterized in that The method further comprises: The second device receives first capability information sent by the first device, where the first capability information is used to indicate one or more of the following: whether the first device supports a model testing function; One or more model identifiers supporting a model testing function in the first device; One or more model identifiers in the first device that do not support a model test function; The number of models supported by the first device for simultaneous model testing; Whether the first device supports reporting first suggestion information, where the first suggestion information indicates a model that the first device expects to suspend testing, or the first suggestion information is used to indicate that the first device expects to suspend all model tests.

45. A communication device, characterized in that: The communication device is a first device, comprising: A receiving unit is used to receive first information sent by a second device, where the first information is used to perform performance testing and / or performance evaluation on a first model that is not activated and used in the first device.

46. ​​The communication device according to claim 45, characterized in that The first information includes one or more of the following: an identifier of the first model; A first parameter, wherein the first parameter is used to determine an activation time of the first model; a second parameter, wherein the second parameter is used to determine whether the inference result of the first model meets the accuracy requirement; a third parameter, the third parameter being used to configure a reference signal for testing the first model; A fourth parameter is associated with the testing process of the first model.

47. The communication device according to claim 46, characterized in that If the first information includes the first parameter, the first parameter is used to configure the first delay information, and the first device activates the first model after a time period corresponding to the first delay information has elapsed since the first device received the first information; or, The first parameter is used to configure the absolute time information for activating the first model.

48. The communication device according to claim 46 or 47, characterized in that If the first information includes the second parameter, and the second parameter includes an accuracy threshold, if the absolute value of the difference between the inference result of the first model and the actual result is less than or equal to the accuracy threshold, it is considered that the inference result of the first model meets the accuracy requirements.

49. The communication device according to any one of claims 46 to 48, characterized in that: If the first information includes the fourth parameter, the fourth parameter is used to configure the running time information of the first timer, and the first timer is associated with the inference result of the first model.

50. The communication device according to claim 49, characterized in that The communication device further comprises: When the first timer times out, the first sending unit is used to send second information to the second device, where the second information includes an identifier of the first model associated with the first timer that has timed out, and the identifier of the first model associated with the first timer indicates the corresponding model The associated first timer has timed out.

51. The communication device according to claim 49 or 50, characterized in that: The communication device further comprises: When the first timer times out, the first processing unit is used to stop running the first model associated with the first timer that has timed out.

52. The communication device according to claim 49, characterized in that The fourth parameter is used to configure the running time information of the second timer, and the second timer is associated with the test result reporting operation of the first model.

53. The communication device according to claim 52, characterized in that The communication device further comprises: During the operation of the second timer, when the first timer times out, the second processing unit is used to record the identifier of the first model associated with the first timer that times out, and stop running the first model associated with the first timer that times out; and / or, When the second timer times out, the second sending unit is used to send second information to the second device, wherein the second information includes an identifier of the first model associated with the first timer that times out, and the identifier of the first model associated with the first timer indicates that the first timer associated with the corresponding model has timed out.

54. The communication device according to claim 52, characterized in that The communication device further comprises: During the operation of the second timer, when the first timer times out, the third processing unit is used to record the identifier of the first model associated with the first timer that times out and the timeout number information of the first timer; and / or, When the second timer times out, the third sending unit is used to send second information to the second device, where the second information includes an identifier of a first model associated with a first timer that has timed out at least once and timeout count information of the first timer.

55. The communication device according to claim 54, characterized in that: If the information entry associated with the first timer that has timed out is not recorded in the cache of the first device, the third processing unit is used to create a new information entry, and record the identifier of the first model associated with the first timer that has timed out in the newly created information entry, and record the timeout number information of the first timer as 1; and / or, If an information entry associated with a first timer that has timed out is recorded in the cache of the first device, the third processing unit is configured to add 1 to the value of the number of stop times of the first timer recorded in the corresponding information entry.

56. The communication device according to any one of claims 49 to 55, characterized in that: The start condition of the first timer includes that the inference result of the first model does not meet the accuracy requirement; and / or, The stopping condition of the first timer includes that the inference result of the first model meets the accuracy requirement.

57. The communication device according to any one of claims 49 to 55, characterized in that: The start condition of the first timer includes that M consecutive inference results of the first model do not meet the accuracy requirement, where M is a positive integer greater than or equal to 1; and / or, The stop condition of the first timer includes that N consecutive inference results of the first model meet the accuracy requirement, where N is a positive integer greater than or equal to 1.

58. The communication device according to any one of claims 46 to 48, characterized in that: The fourth parameter is used to configure the running time information of the second timer, and the second timer is associated with the test result reporting operation of the first model.

59. The communication device according to claim 58, characterized in that The communication device further comprises: When the second timer times out, the fourth sending unit is used to send second information to the second device, where the second information includes the model test results recorded during the operation of the second timer, and the model corresponding to the model test results recorded during the operation of the second timer includes the first model.

60. The communication device according to any one of claims 46 to 48, characterized in that: If the first information includes the fourth parameter, the fourth parameter is used to configure a target inference times threshold for testing the first model.

61. The communication device according to claim 60, characterized in that The communication device further comprises: When the number of inferences for testing the first model reaches the target inference number threshold, the fifth sending unit is used to send second information to the second device, where the second information includes a test result of testing the first model.

62. The communication device according to claim 60, characterized in that The fourth parameter is used to configure the duration of a second timer, and the second timer is associated with the test result reporting operation of the first model.

63. The communication device according to claim 62, characterized in that The communication device further comprises: During the operation of the second timer, when the number of inferences tested on the first model reaches the target inference number threshold, the fourth processing unit is used to stop running the first model whose inference number reaches the target inference number threshold; and / or, When the second timer times out, the sixth sending unit is used to send second information to the second device, where the second information includes the model test results recorded during the operation of the second timer, and the model corresponding to the model test results recorded during the operation of the second timer includes the first model.

64. The communication device according to any one of claims 53 to 55, 59 and 63, characterized in that: The second information includes one or more of the following: one or more model inference results of the first model during the operation of the second timer; an actual result associated with one or more model inference results of the first model during the operation of the second timer; the number of inferences that meet the accuracy requirement among one or more model inference results of the first model during the operation of the second timer; The number of inferences that do not meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer; Proportional information or ratio information between the number of inferences that meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer; The ratio information or value information between the number of inferences that do not meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer.

65. The communication device according to any one of claims 52-55, 58-59 and 62-64, characterized in that: The second timer is associated with one or more of the first models, and the communication device further includes: When the second timer times out, the fifth processing unit is used to stop running one or more of the first models associated with the second timer.

66. Any one of claims 52-55, 58-59 and 62-65, characterized in that The start time of the second timer is determined based on the time when the first device receives the first information, or the start time of the second timer is determined based on the time when the first device activates the first model.

67. The communication device according to any one of claims 45 to 66, characterized in that: The communication device further comprises: When the first condition is met, the sixth processing unit is configured to perform a first operation, where the first operation includes one or more of the following operations: Stop running one or more of the first models being tested in the first device; Stopping a timer associated with the first model test that is running in the first device; Deleting the configuration included in the stored first information; Clear the record information associated with the test process of the first model recorded in the cache of the first device.

68. The communication device according to claim 67, characterized in that The first condition includes one or more of the following: The first device triggers connection re-establishment; The first device triggers a radio link failure; The first device triggers a beam failure; The first device receives synchronization reconfiguration information; The first device performs switching; The first device receives a connection release message; The first device receives a first instruction sent by the second device, where the first instruction instructs the first device to stop testing the first model.

69. The communication device according to any one of claims 45 to 68, characterized in that: The communication device further comprises: The seventh sending unit is used to send first suggestion information to the second device, where the first suggestion information includes a model identifier of the model that the first device expects to suspend testing, or the first suggestion information is used to indicate to the second device that the first device expects to suspend all model tests.

70. The communication device according to any one of claims 45 to 69, characterized in that: The communication device further comprises: An eighth sending unit is configured to send first capability information to the second device, where the first capability information is used to indicate one or more of the following: whether the first device supports a model testing function; One or more model identifiers supporting a model testing function in the first device; One or more model identifiers in the first device that do not support a model test function; The number of models supported by the first device for simultaneous model testing; Whether the first device supports reporting first suggestion information, where the first suggestion information indicates a model that the first device expects to suspend testing, or the first suggestion information is used to indicate that the first device expects to suspend all model tests.

71. A communication device, characterized in that: The communication device is a second device, comprising: A sending unit is used to send first information to a first device, where the first information is used to perform performance testing and / or performance evaluation on a first model that is not activated for use by the first device.

72. The communication device according to claim 71, characterized in that The first information includes one or more of the following: an identifier of the first model; A first parameter, wherein the first parameter is used to determine an activation time of the first model; a second parameter, wherein the second parameter is used to determine whether the inference result of the first model meets the accuracy requirement; a third parameter, the third parameter being used to configure a reference signal for testing the first model; A fourth parameter is associated with the inference result of the first model.

73. The communication device according to claim 72, characterized in that If the first information includes the first parameter, the first parameter is used to configure the first delay information, and the first device activates the first model after a time period corresponding to the first delay information has elapsed since the first device received the first information; or, The first parameter is used to configure the absolute time information for activating the first model.

74. The communication device according to claim 72 or 73, characterized in that If the first information includes the second parameter, and the second parameter includes an accuracy threshold, if the absolute value of the difference between the inference result of the first model and the actual result is less than or equal to the accuracy threshold, it is considered that the inference result of the first model meets the accuracy requirements.

75. The communication device according to any one of claims 72 to 74, characterized in that: If the first information includes the fourth parameter, the fourth parameter is used to configure the running time information of the first timer, and the first timer is associated with the inference result of the first model.

76. The communication device according to claim 75, characterized in that The communication device further comprises: The first receiving unit is used to receive second information sent by the first device, where the second information includes an identifier of a first model associated with a first timer that has timed out, and the identifier of the first model associated with the first timer indicates that a first timer associated with the corresponding model has timed out.

77. The communication device according to claim 75, characterized in that The fourth parameter is used to configure the running time information of the second timer, and the second timer is associated with the test result reporting operation of the first model.

78. The communication device according to claim 77, characterized in that The communication device further comprises: A second receiving unit is used to receive second information sent by the first device, wherein the second information includes an identifier of a first model associated with a first timer that has timed out, and the identifier of the first model associated with the first timer indicates that a first timer associated with the corresponding model has timed out, wherein the second information includes information recorded during the operation of the second timer.

79. The communication device according to claim 77, characterized in that The communication device further comprises: A third receiving unit is used to receive second information sent by the first device, wherein the second information includes an identifier of a first model associated with a first timer that has timed out at least once and timeout information of the first timer, wherein the information included in the second information is recorded during the operation of the second timer.

80. The communication device according to any one of claims 72 to 74, characterized in that: The fourth parameter is used to configure the running time information of the second timer, and the second timer is associated with the test result reporting operation of the first model.

81. The communication device according to claim 80, characterized in that The communication device further comprises: The fourth receiving unit is used to receive second information sent by the first device, where the second information includes a model test result recorded during the operation of the second timer, and the model corresponding to the model test result recorded during the operation of the second timer includes the first model.

82. The communication device according to any one of claims 72 to 74, characterized in that: If the first information includes the fourth parameter, the fourth parameter is used to configure a target inference times threshold for testing the first model.

83. The communication device according to claim 82, characterized in that The communication device further comprises: The fifth receiving unit is used to receive second information sent by the first device, where the second information includes a test result of testing the first model.

84. The communication device according to claim 82, characterized in that The fourth parameter is used to configure the duration of a second timer, and the second timer is associated with the test result reporting operation of the first model.

85. The communication device according to claim 84, characterized in that The communication device further comprises: The sixth receiving unit is used to receive second information sent by the first device, where the second information includes a model test result recorded during the operation of the second timer, and the model corresponding to the model test result recorded during the operation of the second timer includes the first model.

86. The communication device according to any one of claims 78 to 79, 81 and 85, characterized in that: The second information includes one or more of the following: one or more model inference results of the first model during the operation of the second timer; an actual result associated with one or more model inference results of the first model during the operation of the second timer; the number of inferences that meet the accuracy requirement among one or more model inference results of the first model during the operation of the second timer; The number of inferences that do not meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer; Proportional information or ratio information between the number of inferences that meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer; The ratio information or value information between the number of inferences that do not meet the accuracy requirement in one or more model inference results of the first model during the operation of the second timer and the total number of model inferences of the first model during the operation of the second timer.

87. The communication device according to any one of claims 71 to 86, characterized in that: The communication device further comprises: The seventh receiving unit is used to receive first suggestion information sent by the first device, where the first suggestion information includes a model identifier of a model that the first device expects to suspend testing, or the first suggestion information is used to indicate to the second device that the first device expects to suspend all model tests.

88. The communication device according to any one of claims 71 to 87, characterized in that: The communication device further comprises: An eighth receiving unit is configured to receive first capability information sent by the first device, where the first capability information is used to indicate one or more of the following: whether the first device supports a model testing function; One or more model identifiers supporting a model testing function in the first device; One or more model identifiers in the first device that do not support a model test function; The number of models supported by the first device for simultaneous model testing; Whether the first device supports reporting first suggestion information, where the first suggestion information indicates a model that the first device expects to suspend testing, or the first suggestion information is used to indicate that the first device expects to suspend all model tests.

89. A communication device, characterized in that: It comprises a transceiver, a memory and a processor, wherein the memory is used to store programs, and the processor is used to call the programs in the memory and control the transceiver to receive or send signals so that the communication device executes the method as described in any one of claims 1-44.

90. A device, characterized in that The device comprises a processor, configured to call a program from a memory so as to cause the device to execute a method as claimed in any one of claims 1 to 44.

91. A chip, characterized in that: It comprises a processor, which is used to call a program from a memory, so that a device equipped with the chip executes a method as claimed in any one of claims 1 to 44.

92. A computer-readable storage medium, characterized in that A program is stored thereon, the program causing a computer to execute the method according to any one of claims 1 to 44.

93. A computer program product, characterized in that A program is included, which causes a computer to execute the method as claimed in any one of claims 1 to 44.

94. A computer program, characterized in that The computer program causes a computer to execute the method according to any one of claims 1 to 44.

Images