Communication method, device and system

Abstract

A communication method, device and system, relating to the technical field of communications. The method comprises: receiving first information, the first information being used for indicating first activity parameters, the first activity parameters being activity parameters of N first modules in a first model, and N being a positive integer; and on the basis of the first activity parameters, performing inference on the basis of the N first modules. According to the solution, a first device used for inference obtains the activity parameters in full parameters of the first modules by means of the first information, instead of storing the full parameters of the first module. Compared with the solution of locally storing the full parameters in the first device and reading the activity parameters from the full parameters, the described solution can save storage space of the first device, so that the first device having limited storage space can run a large model having massive parameters, thereby improving the inference performance of the large model.

Description

Communication method, device and system

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on March 21, 2024, with application number 202410332551.1 and application name “Communication Methods, Devices and Systems”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of communication technology, and more particularly, to a communication method, device, and system. Background Art

[0003] The booming development of big models has spurred their application across various industries. Cloud-side big models can refer to large models deployed on cloud-side or edge devices. On-device big models can refer to large models deployed on device-side devices. On-device big models are attracting significant attention because they can meet user needs for protecting device-side privacy data. However, the limited storage space on device-side devices can only accommodate large models with smaller parameters. The performance of small-parameter large models is far inferior to that of large-parameter models, hindering their application on the device side.

[0004] Therefore, in the process of inferencing large models on the end, how to deal with the contradiction between the excessive number of model parameters and the limited storage space, and thereby improve the inference performance of large models on the end, is an urgent problem to be solved. Summary of the Invention

[0005] The present application provides a communication method, device, and system that can improve the reasoning performance of large end-side models.

[0006] In the first aspect, a communication method is provided, which can be executed by a first device (for example, a terminal device or an end-side device), or by a component in the first device (for example, a processor, a chip, or a chip system, etc.), or by a logic module or software that can realize all or part of the functions of the first device.

[0007] The method includes: receiving first information, the first information is used to indicate a first activity parameter, the first activity parameter is the activity parameter of N first modules in the first model, N is a positive integer; according to the first activity parameter, reasoning is performed based on the N first modules.

[0008] Through the above embodiment, the first device for reasoning may not store the full parameters of the first module, but obtain the active parameters in the full parameters of the first module through the first information. Compared with the solution of storing the full parameters locally in the first device and reading the active parameters from the full parameters, the above embodiment can save the storage space of the first device, so that the first device with limited storage space can run a large model with a large number of parameters, thereby improving the reasoning performance of the large model. For example, the first module can be a feedforward neural network (FNN) module in a large model of a transformer structure. The ratio of the full parameters of the FNN module to the full parameters of the multi-head attention (MHA) module is approximately 2:1. If the full parameters of the FNN module are not stored in the end-side device, two-thirds of the flash memory space of the end-side device will be saved. The saved flash memory space can store the parameters of other modules, such as the parameters of the MHA module. In this way, the number of parameters of the large model can be increased by three times, thereby effectively improving the performance of the large model.

[0009] In some implementations, the N first modules include N1 first modules, where the N1 first modules are modules of a first layer in at least one layer in the first model, and N1 is a positive integer less than or equal to N.

[0010] Through the above embodiment, the first device can perform reasoning based on the modules in the first layer of the first model based on the activity parameters indicated by the first information. Compared to the solution of reasoning based on the full parameters and the modules in the first layer, the embodiment of the present application does not need to load the full parameters of the modules in the first layer into the memory, thereby saving memory space of the first device and further improving the reasoning performance of the first model.

[0011] In some implementations, the N1 first modules are the second of the multiple modules in the first layer of the first model, N1=1, and the method further includes: predicting information about the position of the first activity parameter based on a second input value, wherein the second input value is the input value of the second of the multiple modules in the first layer of the first model or the output value of the first of the multiple modules in the first layer of the first model.

[0012] Through the above embodiment, the first device can predict the activity parameters of the second of the multiple modules in the first layer of the first module, and thus perform reasoning based on the second of the multiple modules according to the activity parameters of the second of the multiple modules. Compared with the solution of reasoning based on the full parameters and the second of the multiple modules in the first layer, the embodiment of the present application does not need to load the full parameters of the second of the multiple modules in the first layer into the memory, thereby saving memory space of the first device, thereby further improving the reasoning performance of the first model.

[0013] In some implementations, the method further includes: predicting information about the position of a second activity parameter based on the first input value, the second activity parameter being an activity parameter of the first of multiple modules in the first layer of the first model, and the first input value being the input value of the first of multiple modules in the first layer of the first model.

[0014] Through the above embodiment, the first device can predict the activity parameters of the first one in the first layer of the first module, and thus perform reasoning based on the first one of the multiple modules according to the activity parameters of the first one of the multiple modules. Compared with the solution of reasoning based on the full parameters and the first one of the multiple modules in the first layer, the embodiment of the present application does not need to load the full parameters of the first one of the multiple modules in the first layer into the memory, thereby saving the memory space of the first device, thereby further improving the reasoning performance of the first model.

[0015] In some implementations, the N first modules further include N2 first modules, the first model includes N2 layers, each of the N2 layers includes one first module among the N2 first modules, N2≥2, and N2 is a positive integer less than or equal to N.

[0016] Through the above embodiment, the second device can send activity parameters of multiple layers at one time, reducing the number of times the activity parameters are sent, thereby reducing the signaling overhead caused by frequent sending of activity parameters.

[0017] In some implementations, the first model includes M layers, where M is a positive integer greater than N2, and the N2 layers are not numbered consecutively among the M layers.

[0018] Through the above embodiment, the first activity parameter indicated by the first information may include activity parameters of discontinuously numbered layers, that is, these activity parameters may skip one or more layers, thereby reducing the amount of information transmitted in the first information and lowering the delay.

[0019] In some implementations, N2 is related to resources and / or inferred latency requirements for transmitting the first information.

[0020] Through the above embodiment, the value of N2 can be related to the resources used to transmit the first information and / or the inference latency requirement, so that the transmission of the first information can meet the resource constraints and / or inference latency requirements. The above solution can ensure that the first information can be transmitted smoothly, or can ensure that the transmission of the first information can meet the inference latency requirements of the first model, thereby further ensuring the inference performance of the first model.

[0021] In some implementations, the method further includes: predicting information about the position of the first activity parameter based on an input value of a layer previous to the first layer in the N2 layers.

[0022] Through the above embodiment, on the one hand, the first device can predict the information of the positions of the activity parameters of the multiple first modules respectively included in the multiple layers. Compared with the scheme of predicting the activity parameters of only one module at a time, the above embodiment can save computing resources. On the other hand, the first device predicts N2 layers based on the input value of the previous layer of the first layer in the N2 layers, rather than based on the input value of the first layer in the N2 layers. In this way, the first device can predict the information of the positions of the activity parameters of the first modules in the N2 layers when inferring the previous layer of the first layer in the N2 layers. That is to say, the layer for which the first device predicts the position of the activity parameters is not the layer being inferred, so the above embodiment will not affect the layer being inferred, that is, it will not reduce the inference performance of the large model.

[0023] In some implementations, each of the N2 layers also includes a second module, which is one of the N2 second modules in the first model. The method also includes: predicting information about the position of a third activity parameter based on the input value of the previous layer of the first layer in the N2 layers, where the third activity parameter is the activity parameter of the N2 second modules.

[0024] Through the above embodiment, on the one hand, the first device can predict the location information of the activity parameters of multiple second modules included in multiple layers. Compared with a solution that predicts the activity parameters of only one module at a time, this embodiment can save computing resources. On the other hand, the layer for which the first device predicts the location of the activity parameters is not the layer currently being inferred. Therefore, this embodiment does not affect the layer currently being inferred, that is, it does not reduce the inference performance of large models.

[0025] In some implementations, the second module is an MHA module.

[0026] In some implementations, each of the N2 layers also includes a second module, and the method further includes: determining a third input value based on the output value of the first module in the k-th layer among the N2 layers and the output value of the second module in the k-th layer, the third input value being used as the input of the first module in the next layer of the k-th layer and the input of the second module in the next layer of the k-th layer, where k is a positive integer less than or equal to N2.

[0027] Through the above embodiment, one or more layers in the N2 layers can adopt the form of parallel reasoning, that is, when the first module of a layer is performing reasoning, the second module of this layer can also perform reasoning. Sequential reasoning is to use the output obtained after the second module of a layer completes reasoning as the input of the second module of this layer. Compared with the layer using sequential reasoning, the number of inputs of the layer using parallel reasoning can be reduced from 2 to 1. The number of inputs is reduced, and the rate of change of adjacent inputs is constant, which can increase the value of N2 that can be selected, so that the activity parameters indicated by the first information sent by the second device can be the activity parameters of more layers. In other words, the second device can send the activity parameters of more layers at one time, thereby further reducing the transmission delay from the second device to the first device.

[0028] In some implementations, the method further includes: when the confidence level of the output value of the first model is greater than or equal to a first threshold, sending second information, where the second information is used to instruct to stop transmitting the first information.

[0029] Through the above embodiment, when the confidence level of the output value of the first model meets the conditions, the first device can send the second information, and the second information can indicate to stop transmitting the first information, thereby reducing the waste of resources caused by transmitting the first information. In addition, the above embodiment can be applied to the scenario of early exiting of reasoning. In the scenario where the first model exits reasoning early, the exit of the first model from reasoning is not due to the completion of reasoning for all layers of the first model, but because the confidence level of the output value of the first model meets the conditions. Early exit from reasoning can prevent some layers in the first model from participating in reasoning, thereby reducing the reasoning delay. The above embodiment can avoid the second device from transmitting information indicating active parameters used for reasoning when the first model exits reasoning early, thereby saving transmission resources.

[0030] In some implementations, the method further includes: sending third information, the third information including information indicating a location of the first activity parameter, the third information being used to request the first activity parameter.

[0031] Through the above embodiment, the first device can request to send the first activity parameter, and the request sent by the first device can carry information indicating the location of the first activity parameter, so that the value of the activity parameter indicated by the first information is accurate.

[0032] In some implementations, the first module is a FNN module.

[0033] On the second aspect, a communication method is provided, which can be executed by a second device (for example, a network device, a core network device, a cloud-side device or an edge-side device), or by a component in the second device (for example, a processor, a chip, or a chip system, etc.), or by a logic module or software that can realize all or part of the functions of the second device.

[0034] The method includes: generating first information, the first information is used to indicate a first activity parameter, the first activity parameter is the activity parameter of N first modules in the first model, the first activity parameter is used to perform inference based on the N first modules, N is a positive integer; sending the first information.

[0035] In some implementations, the N first modules include N1 first modules, where the N1 first modules are modules of a first layer in at least one layer in the first model, and N1 is a positive integer less than or equal to N.

[0036] In some implementations, the N first modules further include N2 first modules, the first model includes N2 layers, each of the N2 layers includes one first module among the N2 first modules, N2≥2, and N2 is a positive integer less than or equal to N.

[0037] In some implementations, the first model includes M layers, where M is a positive integer greater than N2, and the N2 layers are not numbered consecutively among the M layers.

[0038] In some implementations, N2 is related to resources and / or inferred latency requirements for transmitting the first information.

[0039] In some implementations, the method further includes: receiving second information, where the second information is used to instruct to stop transmitting the first information.

[0040] In some implementations, the method further includes receiving third information, where the third information includes information indicating a location of the first activity parameter.

[0041] In some implementations, sending the first information includes sending the first information in response to the third information.

[0042] In some implementations, the first module is a FNN module.

[0043] In a third aspect, a communication device is provided, comprising a processing circuit (or processor) and an input / output interface (also referred to as an interface circuit), the input / output interface being used to input and / or output signals, the processing circuit being used to execute the first aspect and any possible method of the first aspect, or the processing circuit being used to execute the second aspect and any possible method of the second aspect.

[0044] In some implementations, the processing circuit is used to communicate with other devices through the interface circuit and execute the above-mentioned first aspect and any possible method of the first aspect, or execute the second aspect and any possible method of the second aspect.

[0045] In a fourth aspect, a communication device is provided, which may include a unit or module for performing the functions of the communication device.

[0046] In some implementations, the communication device may include modules, units, or means corresponding to the methods / operations / steps / actions described in the first aspect and any possible implementation of the first aspect. The modules, units, or means may be hardware circuits, software, or a combination of hardware circuits and software.

[0047] In some implementations, the communication device may include modules, units, or means for executing the methods / operations / steps / actions described in the second aspect and any possible implementation of the second aspect. The modules, units, or means may be hardware circuits, software, or a combination of hardware circuits and software.

[0048] In a fifth aspect, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or the instruction is run on a computer, the first aspect and any possible method of the first aspect are executed, or the second aspect and any possible method of the second aspect are executed.

[0049] In the sixth aspect, a computer program product is provided, comprising a computer program or instructions, which, when run on a computer, causes the first aspect and any possible method of the first aspect to be executed (or implemented), or causes the second aspect and any possible method of the second aspect to be executed (or implemented).

[0050] In the seventh aspect, a communication device is provided, comprising a processor, for causing the device to execute any possible method of the first aspect, or causing the device to execute any possible method of the second aspect, by executing a computer program (or computer executable instructions) stored in a memory, and / or through a logic circuit.

[0051] In one possible implementation, the device further includes a memory. In one possible implementation, the processor and the memory are integrated together. In another possible implementation, the memory is located outside the communication device. The processor may include one or more.

[0052] In one possible implementation, the communication device further includes a communication interface, which is used for the communication device to communicate with other devices, such as sending or receiving data and / or signals. Exemplarily, the communication interface can be a transceiver, circuit, bus, module, or other type of communication interface.

[0053] In one implementation, the communication device of the third aspect, fourth aspect or seventh aspect may be a chip or a chip system.

[0054] In an eighth aspect, a chip is provided, comprising a processor for calling a computer program or computer instruction in a memory so that the processor executes any one of the implementation methods of the above-mentioned first aspect, or so that the processor executes any one of the implementation methods of the above-mentioned second aspect.

[0055] In some implementations, the processor is coupled to the memory through an interface.

[0056] In the ninth aspect, a communication system is provided, comprising a first device and a second device, wherein the first device is used to execute the above-mentioned first aspect and any possible implementation method of the first aspect, and the second device is used to execute the above-mentioned second aspect and any possible implementation method of the second aspect.

[0057] The description of the advantageous effects of any of the second to ninth aspects etc. may refer to the description of the advantageous effects of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] FIG1 is a schematic diagram of a communication system.

[0059] FIG2 is a schematic block diagram of another communication system.

[0060] FIG3 is a schematic diagram of an inference method.

[0061] FIG4 is a schematic flowchart of a communication method provided in an embodiment of the present application.

[0062] FIG5 is a schematic flowchart of another communication method provided in an embodiment of the present application.

[0063] FIG6 is a schematic flowchart of another communication method provided in an embodiment of the present application.

[0064] FIG7 is a schematic diagram of another communication method provided in an embodiment of the present application.

[0065] FIG8 is a schematic flowchart of another communication method provided in an embodiment of the present application.

[0066] FIG9 is a schematic diagram of a large model for parallel reasoning provided in an embodiment of the present application.

[0067] FIG10 is a schematic block diagram of a communication device according to an embodiment of the present application.

[0068] FIG11 is a schematic block diagram of another communication device according to an embodiment of the present application. DETAILED DESCRIPTION

[0069] The technical solution in this application will be described below with reference to the accompanying drawings.

[0070] This application will present various aspects, embodiments, or features in the context of systems that may include multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. Furthermore, combinations of these aspects may also be used.

[0071] Additionally, in the embodiments of this application, words such as "exemplary" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.

[0072] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field will know that with the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0073] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically stated. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically stated.

[0074] The first, second, etc. descriptions appearing in the embodiments of the present application are, unless otherwise specified, only used for illustration and distinction of the description objects. There is no order, nor does it indicate any special limitation on the number in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0075] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0076] It should be understood that the term "and / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0077] To facilitate understanding of this application, the following provides exemplary explanations of terms that may appear in this application.

[0078] MHA block: The MHA block can be a multi-head attention mechanism module in the transformer structure of a large model. The large model can be an artificial intelligent (AI) model or a machine learning (ML) model.

[0079] FNN module (block): The FNN module can be a feedforward neural network module placed after the MHA module in the transformer structure of the large model. The input of the FNN module can be the output of the MHA module.

[0080] Transformer layer: A transformer layer may include an MHA module and at least one FNN module in sequence. The output of the MHA module in a layer can be used as the input of the FNN module in this layer. A large model may include a transformer structure; a transformer structure includes one or more transformer layers. The following example uses a transformer layer including an MHA module and an FNN module, but it should be noted that this application is also applicable to the case where a transformer layer includes multiple MHA modules and multiple FNN modules.

[0081] Input: Input can be the input of the large model, or the input to the first of the multiple modules in the first layer of the large model. For example, the input can be a vector or tensor after the user's request to the large model is embedded. For another example, the input can be data that the large model is triggered by an event or regularly obtains from the user side (such as the end-side device). The input can also be the input to a block in the middle of the large model (for example, an FNN module or an MHA module). For example, the input can be the vector or tensor input to the second of the multiple modules in the first layer of the large model. For another example, the input can be a vector or tensor input to a module in the second layer of the large model or any layer after the second layer.

[0082] Prediction module (predictor): A predictor can be an AI model or an ML model. For example, a predictor can be a small, trained deep-learning neural network (DNN). The predictor can predict the location of the active parameters of the MHA module or FNN module based on the input.

[0083] Activated parameter: The activated parameter may include the weight parameter of the activated neuron and / or the weight parameter corresponding to the activated head.

[0084] Active Neurons: The output of an FNN module can be represented as σ(xW+b). σ represents the activation function, for example, a rectified linear unit (ReLU); x represents the input to the FNN module; W represents the weight parameter (for example, a weight matrix); and b represents the bias parameter (for example, a bias matrix). For an FNN module, the active parameter can be the weight parameter of the active neuron (or the element in the weight matrix). Active neurons are neurons whose weight parameters (or neuron weight parameters) exceed a preset threshold when their inner product with the input of the FNN module exceeds a preset threshold. For example, if an input x1 is calculated based on x1W+b and the result is a negative number or a very small positive number, that is, below a preset threshold, then after further processing by the activation function, the output value σ(x1W+b) has little effect on the final output of the FNN module. Therefore, the neurons used to calculate x1 can be inactive. The corresponding elements in matrix W for these inactive neurons can be set to 0, resulting in matrix W'. The matrix W' can be a sparse matrix with most elements being zero. Reasoning based on the matrix W' can save computing resources and reduce reasoning latency. For example, if an input x2 is calculated based on x2W+b, and the result exceeds the preset threshold, then it means that after the result is further processed by the activation function, the output value σ(x2W+b) obtained has a certain impact on the final output result of the FNN module. Therefore, the relevant neurons used to calculate the x2 are active neurons. The elements corresponding to the above-mentioned active neurons in the matrix W can remain unchanged, so that when the processed matrix W' is reasoned again, the above-mentioned active neurons can participate in the reasoning.

[0085] Active head: Combines the input of the MHA module with the weight parameters (such as weight matrix) W q 、W k and W v Multiply them respectively to get the three matrices Q, K and V. Assuming that the number of heads of the MHA module is n, the matrix Q can be divided into n sub-matrices, namely Q1 to Q n , where each submatrix corresponds to a head, and n is a positive integer. For example, if the matrix Q is a 512*1 (512 rows and 1 column) matrix, and n=8, the matrix Q can be divided into 8 64*1 submatrices, namely submatrices Q1 to Q8. Submatrices Q1 to Q8 correspond to 8 heads respectively. Similarly, the matrix K can be divided into submatrices K1 to K n , where each submatrix corresponds to a head; the matrix V can be divided into submatrices V1 to V n, where each submatrix corresponds to a head. Thus, each head corresponds to three submatrices, namely Q i , K i and V i , i is an integer from 1 to n. Each head is based on the input and the three sub-matrices Q corresponding to the head i , K i and V i Perform inference to obtain the output of the head. Perform inner product between the input and the output of the head. If the result of the inner product exceeds the preset threshold, it means that the head is an active head, otherwise it is an inactive head. For the MHA module, the activity parameter can be the weight matrix corresponding to at least one active head. For example, perform inner product between the input and the output of the head. If the result is larger, such as exceeding the preset threshold, the three sub-matrices Q corresponding to the head can be i , K i and V i Retain; if the obtained head parameter (or the output result of the head) is small, for example, lower than the preset threshold, the three sub-matrices Q corresponding to the head can be i , K i and V i Set to 0. In this way, for an MHA module including multiple heads, the sub-matrices of some heads in the multiple heads may be set to 0, thereby affecting the initial weight matrix W q 、W k and W v The value of , the weight matrix W after the change is obtained q '、W k ' and W v '. The changed weight matrix W q '、W k ' and W v ' can be called the weight matrix corresponding to at least one active head.

[0086] It should be noted that the above “exceeds the preset threshold” may refer to being greater than or equal to the preset threshold, or greater than the preset threshold. The above “below the preset threshold” may refer to being less than the preset threshold, or less than or equal to the preset threshold.

[0087] The technical solutions of the embodiments of the present application can be applied to various communication systems, including but not limited to: Long Term Evolution (LTE) system, New Radio (NR) system and other fifth generation (5G) systems. thgeneration (5G) mobile communication systems, narrowband internet of things (NB-IoT) systems, enhanced machine-type communication (eMTC) systems, enhanced mobile broadband (eMBB) systems, ultra-reliable low latency communications (URLLC) systems, satellite communication systems, LTE-machine-to-machine (LTE-M) systems, or sixth generation (6 th generation, 6G) mobile communication systems and other systems that have evolved after 5G.

[0088] It should be noted that in the embodiments of the present application, the term "communication" can also be described as "data transmission," "signal transmission," "information transmission," or "transmission." In the embodiments of the present application, transmission can include sending or receiving. For example, transmission can be uplink transmission, for example, a terminal device sending a signal to a network device; downlink transmission, for example, a network device sending a signal to a terminal device; or sideways transmission, for example, a terminal device sending a signal to another terminal device.

[0089] Figure 1 is a schematic diagram of a communication system 100. As shown in Figure 1 , the communication system 100 includes a radio access network 110 and a core network 120. Optionally, the communication system 100 may also include the Internet 130. The radio access network 110 may include at least one network device (such as 111a and 111b in Figure 1 ) and at least one terminal device (such as 112a-112j in Figure 1 ). The terminal device is wirelessly connected to the network device. The network device is wirelessly or wiredly connected to the core network 120. The core network 120 may include one or more core network devices. The core network devices and the network devices may be independent, distinct physical devices, or they may integrate the functions of the core network device and the logical functions of the network device into the same physical device. Alternatively, a single physical device may integrate some of the functions of the core network device and some of the functions of the network device. Terminal devices and network devices may be connected to each other via wired or wireless connections. Terminal devices, network devices, and terminal devices may communicate wirelessly using air interface resources. Exemplarily, air interface resources may include at least one of time domain resources, frequency domain resources, code resources, and space resources. It should be noted that FIG1 is merely a schematic diagram, and the communication system 100 may also include other network devices, such as wireless relay devices and wireless backhaul devices, which are not shown in FIG1 .

[0090] A network device may be any device with wireless transceiver functions. For example, a network device may be a base station for accessing a terminal device to a radio access network (RAN). A network device may sometimes also be referred to as an access network device or an access network node. It is understandable that in systems using different wireless access technologies, the names of devices with network device functions may be different. For ease of description, the embodiments of the present application will collectively refer to devices that provide wireless communication access functions for terminal devices as base stations. In the embodiments of the present application, network devices include but are not limited to: various forms of macro base stations (111a in Figure 1), micro base stations or indoor stations (111b in Figure 1), pico base stations, small stations, balloon stations, relay stations, access points, etc. Network devices may include evolved node Bs (eNBs or eNodeBs) in LTE, access points (APs) in wireless fidelity (WiFi) systems, wireless relay nodes, wireless backhaul nodes, transmission points (TPs), or transmission reception points (TRPs). They may also include next-generation NodeBs (gNBs) or transmission points (TRPs or TPs) in 5G systems, one or a group of antenna panels (including multiple antenna panels) of a base station in a 5G system, network nodes constituting a gNB or transmission point, such as a baseband unit (BBU) or distributed unit (DU), and network devices, servers, or vehicle-mounted devices in networks evolved beyond 5G, such as 6G. A network device may also be a module or unit that performs some of the functions of a base station, for example, a centralized unit (CU) or a DU.

[0091] In the embodiments of the present application, the apparatus for implementing the function of the network device may be the network device, or may be an apparatus capable of supporting the network device in implementing the function, such as a chip system, which may be installed in the network device. The chip system may be composed of a chip or may include a chip and other discrete components.

[0092] In another possible scenario, multiple network devices collaborate to assist the terminal in achieving wireless access, and different network devices respectively implement part of the functions of the base station. For example, the network device can be a CU, DU, CU-control plane (CP), CU-user plane (UP), or radio unit (RU). The CU and DU can be set separately, or they can be included in the same network element, such as the BBU. The RU can be included in a radio frequency device or radio frequency unit, such as a remote radio unit (RRU), an active antenna unit (AAU), or a remote radio head (RRH).

[0093] In different systems, CU (or CU-CP and CU-UP), DU or RU may also have different names, but those skilled in the art can understand their meanings. For example, in an open radio access network (O-RAN) system, CU may also be called O-CU (open CU), DU may also be called O-DU, CU-CP may also be called O-CU-CP, CU-UP may also be called O-CU-UP, and RU may also be called O-RU. For the convenience of description, this application takes CU, CU-CP, CU-UP, DU and RU as examples for description. Any unit of CU (or CU-CP, CU-UP), DU and RU in this application may be implemented by a software module, a hardware module, or a combination of a software module and a hardware module. The embodiments of this application do not limit the specific technology and specific device form adopted by the network device.

[0094] A terminal device can be a device that provides voice and / or data connectivity to a user; a terminal device can also be a device with wireless connection capabilities. Terminal devices can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted; can also be deployed on the water (such as ships); and can also be deployed in the air (for example, on airplanes, balloons, and satellites). A terminal device can also be called user equipment (UE), access terminal, terminal, subscriber unit (subscriber unit), user station, mobile station, mobile station (MS), mobile terminal (MT), remote station, remote terminal, mobile device, user terminal, wireless network device, user agent, or user device. In the embodiments of the present application, the terminal device includes, but is not limited to, a cellular phone, a mobile phone, a wireless data card, a wireless modem, a tablet computer, a laptop computer, a notebook computer, a PDA, a mobile internet device (MID), a computer with wireless transceiver function, a cordless phone, a session initiation protocol (SIP) phone, a smart phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device (handset) with wireless communication function, a computing device or other device connected to a wireless modem, an in-vehicle device (e.g., a car, a bicycle, an electric car, an airplane, a ship, a train, a high-speed rail, etc.), a wearable device (e.g., a smart watch, a smart bracelet, a pedometer, smart glasses, etc.), a satellite terminal, a terminal device in the Internet of Things or the Internet of Vehicles, and any form of terminal in a future network, a relay user device, or a terminal in a future evolved public land mobile communication network (PLMN), etc.The terminal device may also be a virtual reality (VR) device, an augmented reality (AR) device, a smart point of sale (POS) machine, customer-premises equipment (CPE), a light UE, a reduced capability UE (REDCAP UE), a machine type communication (MTC) terminal, a terminal device in industrial control, a terminal device in self-driving, a terminal device in remote medical, a terminal device in a smart grid, a wireless terminal in transportation safety, a terminal device in a smart city, a terminal device in a smart home, a tactile terminal device, a smart home device (for example, a refrigerator, a television, an air conditioner, an electric meter, etc.), an intelligent robot, a robotic arm, workshop equipment, a wireless terminal in self-driving, or a flying device (for example, an intelligent robot, a hot air balloon, a drone, an airplane), etc. The terminal device may also be a vehicle device, such as a complete vehicle device, an onboard module, an onboard chip, an onboard unit (OBU), or a telematics box (T-BOX). The terminal device may also be other devices with terminal functions, for example, a terminal device may also be a device that functions as a terminal in device-to-device (D2D) communication. The embodiments of the present application are not limited to this.

[0095] In the embodiments of the present application, the device for realizing the function of the terminal device may be a terminal device, or a device capable of supporting the terminal device to realize the function, such as a chip or a chip system, which may be installed in the terminal device. The chip system may be composed of chips, or may include chips and other discrete devices. In the technical solutions of the embodiments of the present application, the device for realizing the function of the terminal device is a terminal device, which may also be referred to as a terminal. The following may take the terminal device as an example to describe the technical solutions provided by the embodiments of the present application.

[0096] The roles of base stations and terminals can be relative. For example, the helicopter or drone 112i in Figure 1 can be configured as a mobile base station. For terminals 112j accessing the wireless access network 110 via 112i, terminal 112i is a base station. However, for base station 111a, 112i is a terminal, meaning that communication between 111a and 112i occurs via a wireless air interface protocol. Of course, communication between 111a and 112i can also occur via a base station-to-base station interface protocol. In this case, 112i is also a base station relative to 111a. Therefore, base stations and terminals can be collectively referred to as communication devices. 111a and 111b in Figure 1 can be referred to as communication devices with base station functionality, and 112a-112j in Figure 1 can be referred to as communication devices with terminal functionality.

[0097] Network devices and terminal devices can communicate via wireless links. The transmission link from a network device to a terminal device is called a downlink (DL) or a downlink channel, used to transmit downlink signals. The transmission link from a terminal device to a network device is called an uplink (UL) or an uplink channel, used to transmit uplink signals.

[0098] Fig. 2 is a schematic block diagram of another communication system 200. The communication system 200 may also be referred to as a cloud system or have other names.

[0099] The communication system 200 includes a central cloud 210, an edge cloud 220 and an end side 230. Among them, the edge cloud 220 may be different from the central cloud 210 (or a traditional cloud platform). For example, the central cloud 210 (or a traditional cloud platform) may include a data center with scaled resources and centralized locations, while the edge cloud 220 may include multiple edge nodes distributed in different areas, wherein the resource scale of a single edge node is smaller. The network covered by the edge cloud 220 is wider and closer to the end side devices in the end side 230. The end side devices in the end side 230 and the edge nodes in the edge cloud 220 can communicate through a routing system or other means.

[0100] Edge nodes can provide various resources to the outside world, for example, computing resources with computing capabilities such as central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU); for example, storage resources with storage functions such as memory and hard disk, as well as network resources such as bandwidth. The types and quantities of resources included in two edge nodes among multiple edge nodes can be the same or different. Multiple edge nodes can be deployed by the same Internet service provider or provided by different Internet service providers. The edge nodes in the edge cloud 220 can be deployed with services that provide cloud computing, that is, applications that can provide services are deployed on the edge nodes. For example, large model functions can be deployed in edge nodes.

[0101] The central cloud 210 may also be referred to as a cloud-side device. The edge cloud 220 or the edge nodes therein may also be referred to as edge-side devices. For example, the end-side device may be a terminal device, such as the terminal device shown in FIG1 .

[0102] In some possible implementations, the central cloud 210 may be deployed in the core network 120. In other words, the core network 120 may have the functionality of the central cloud 210. In some possible implementations, the edge cloud 220 may be deployed in a network device. In other words, the network device may include an edge node.

[0103] The booming development of big models has spurred their application across various industries. Cloud-side big models can refer to large models deployed on cloud-side or edge devices. On-device big models can refer to large models deployed on device-side devices. On-device big models are attracting significant attention because they can meet user needs for protecting device-side privacy data. However, the limited storage space on device-side devices can only accommodate large models with smaller parameters. The performance of small-parameter large models is far inferior to that of large-parameter models, hindering their application on the device side.

[0104] Figure 3 is a schematic diagram of an inference method. The large model shown in Figure 3 can be deployed on the client side and can use a transformer structure.

[0105] The right part of Figure 3 is the legend of the left part. The left part of Figure 3 shows the reasoning process of the first three layers of the large model. Each layer is represented by a short-line rounded rectangle, and each layer includes an MHA module and an FNN module. For example, the MHA module may include 4 heads, which can be represented by 4 rectangles in Figure 3; the input of the MHA module is represented by a dashed line arrow in Figure 3. The FNN module may include 5 neurons, which are represented by 5 squares; the input of the FNN module is represented by a dotted line arrow. The prediction module can predict the position of the active parameters of the MHA module or the FNN module based on the input, and the prediction process is represented by an arc-shaped solid arrow. P1 to P4 in Figure 3 represent the first to fourth predictions of the prediction module, respectively. For example, the first prediction is a prediction of the active parameters of the MHA module of the second layer based on the output of the FNN module of the first layer.

[0106] Specifically, the initial input can be the input to the first-layer MHA module. The device can infer the user input based on the full set of parameters and the first-layer MHA module to obtain an output result. That is, although not all heads are active in the first-layer MHA module, the first-layer MHA module needs to load the submatrices of all heads into the device's memory when performing inference. The output of the first-layer MHA module can serve as the input to the first-layer FNN module. The device can infer the input based on the full set of parameters and the first-layer FNN module to obtain an output result. That is, although not all neurons are active in the first-layer FNN module, the first-layer FNN module needs to load all weight parameters into the device's memory when performing inference. While the first-layer FNN module is performing inference, the prediction module can predict the position of the active parameters of the second-layer MHA module based on the input from the first-layer FNN module, as represented by the curved solid arrows associated with P1 in Figure 3.

[0107] The output of the first-layer FNN module can serve as the input to the second-layer MHA module. Based on the location of the activity parameters of the second-layer MHA module predicted by the prediction module, the device retrieves the activity parameters from its flash memory. The device then infers the input based on the activity parameters and the second-layer MHA module to obtain the output. Specifically, in the second-layer MHA module, only the activity parameters of the active heads are loaded into memory. For example, the first and third heads from the left in Figure 3 are the active heads of the second-layer MHA module, and the corresponding weight matrices are the activity parameters. Furthermore, because the device cannot predict which parameters in the full set of parameters are active, the full set of MHA module parameters must be stored in the device's flash memory so that the active parameters within the full set can be determined based on their location. While the second-layer MHA module is performing inference, the prediction module can predict the activity parameters of the second-layer FNN module based on the input from the second-layer MHA module, as indicated by the curved solid arrows between P2 and P2 in Figure 3. The output of the second-layer MHA module can serve as the input to the second-layer FNN module. Based on the location of the activity parameters of the second-layer FNN module predicted by the prediction module, the end-side device can retrieve the activity parameters from the end-side device's flash memory and perform inference on the input of the second-layer FNN module based on the activity parameters to obtain the output result. That is, in the second-layer FNN module, only the weight parameters of the active neurons are loaded into memory. For example, the neurons in the third and fifth positions from the left in Figure 3 are active neurons, and the corresponding elements in the weight matrix of these two neurons are the activity parameters. Furthermore, because the end-side device cannot predict which parameters in the full set of parameters are active, the end-side device's flash memory needs to store the full set of FNN module parameters so that the active parameters in the full set of parameters can be determined based on the location of the activity parameters. While the second-layer FNN module is performing inference, the prediction module can predict the activity parameters of the third-layer MHA module based on the input of the second-layer FNN module, as represented by P3 and the curved solid arrows associated with P3 in Figure 3. The description of the third-layer MHA module and FNN module refers to the description of the second layer above and is not repeated here.

[0108] The end-side device often has a small flash memory space. According to the scheme shown in Figure 3, the end-side device needs to read the required active parameters from the flash memory during the inference process. Therefore, the end-side device needs to store all the weight matrices of the MHA module and all the weight parameters of the FNN module in the flash memory. Due to the limitation of flash memory space, the end-side device can only store large models with a small number of parameters. For example, a large model with 7 billion (B) parameters is stored in single-precision floating point, which requires 7B×32 bits (bit) / 8 bits per byte (byte) = 28 gigabytes (GB) of flash memory space. The end-side device cannot store large models with a large number of parameters. For example, the end-side device cannot store large models with more than 10 billion parameters. It can be understood by those skilled in the art that the performance of large models with small parameters is lower than that of large models with large parameters, which makes the end-side large model unable to achieve performance similar to that of the cloud-side large model, thereby affecting the user experience.

[0109] End-side devices often have smaller memory space. According to the scheme shown in Figure 3, the first transformer layer uses the full parameters for inference. For example, the generative pre-trained transformer (GPT) 3 includes 96 transformer layers, and the full parameters of each transformer layer occupy a space of 175B / 96x 4=7.3GB. In other words, when GPT3 performs inference on the first transformer layer, it needs to load all 7.3GB of parameters into the memory. However, most end-side devices with limited memory space find it difficult to cope with such high memory usage. For example, a mobile phone may only have 8GB of memory. The limited memory space leads to reduced performance of large models on the end side, making it impossible for large models on the end side to achieve performance similar to that of large models on the cloud side, thus affecting the user experience.

[0110] Therefore, in the process of inferencing large models on the end, how to deal with the contradiction between the excessive number of model parameters and the limited storage space, and thereby improve the inference performance of large models on the end, is an urgent problem to be solved.

[0111] FIG4 is a schematic flow chart of a communication method 400 provided in an embodiment of the present application. Method 400 can improve the inference performance of a large model on the terminal side. Method 400 is described below with reference to FIG4.

[0112] S430: The first device receives first information from the second device. Correspondingly, the second device sends the first information to the first device.

[0113] For example, the first device may be a terminal device or a terminal-side device. In S430, the first information may be received by the first device, or by a component in the first device (e.g., a processor, a chip, or a chip system), or by a logic module or software that implements all or part of the functions of the first device.

[0114] For example, the second device may be a network device, a core network device, a cloud-side device, or an edge device. In S430, the first information may be sent by the second device, or by a component in the second device (e.g., a processor, a chip, or a chip system), or by a logic module or software that implements all or part of the functions of the second device.

[0115] Similar to the description above, other operations involving the first device and the second device in this application can also be performed by the device itself, components in the device, or logical modules or software that can realize all or part of the functions of the device, and will not be repeated here.

[0116] Optionally, the first information is used to indicate a first activity parameter. The first activity parameter may be an activity parameter of N first modules in the first model, where N is a positive integer.

[0117] Among them, the first model can be any large model, AI model or ML model. For example, the first model can be GPT1, GPT2 or GPT3, etc. The first module can be any module in the first model. For example, the first module can be an FNN module. Thus, the first activity parameter can be the activity parameter of the FNN module. However, this application is not limited to this. For example, the first module can also be an MHA module or other module. In this way, the first activity parameter can be the activity parameter of the MHA module or other modules. The first model can include N first modules. In other words, the first model can include one or more first modules. Thus, the first activity parameter can be the activity parameter of one or more modules.

[0118] This application does not limit the name of the first model, which may also be called an AI model, an ML model, or other names. This application does not limit the name of the first activity parameter, which may also be called a parameter, an active neuron, an activity weight, an active element, or other names.

[0119] It should be noted that the present application does not limit the position of the first module, or in other words, does not limit the number of layers in which the first module is located. For example, the first module may be in the first layer. For another example, the first module may be in the third layer. For another example, the three first modules may be in the second layer, the third layer, and the fourth layer, respectively. Therefore, the present application does not limit which layer or layers the first activity parameter is the activity parameter of. For example, the first activity parameter may be the activity parameter of the module of the first layer (referred to as the activity parameter of the first layer, and other descriptions are similar). For another example, the first activity parameter may be the activity parameter of the third layer. For another example, the first activity parameter may include the activity parameter of the second layer, the activity parameter of the third layer, and the activity parameter of the fourth layer.

[0120] This application does not limit the manner in which the first information indicates the first activity parameter. As an example, the first information may include the first activity parameter, so that the first device can obtain the first activity parameter based on the content of the first information. As another example, the first information may include an identifier of the first activity parameter. The first device can pre-acquire the correspondence between multiple activity parameters and multiple identifiers, for example, a mapping table of activity parameters and identifiers. The first device can determine that the activity parameter corresponding to the identifier is the first activity parameter based on the identifier of the first activity parameter and the pre-acquired correspondence, thereby obtaining the first activity parameter.

[0121] This application does not limit the name of the first information. The first information may also be called activity parameter information or have other names.

[0122] In some possible implementations, method 400 further includes: (S420) the second device generates the first information. In some possible implementations, S420 is performed before S430. In some possible implementations, S420 can be replaced by: the second device obtains the first information. For example, the second device is a radio frequency chip in a base station, and the radio frequency chip can obtain the first information from a baseband chip in the base station.

[0123] This application does not limit the manner in which the second device generates the first information. As an example, the second device may pre-store all or part of the activity parameters. The second device may determine the first activity parameter based on the pre-stored activity parameters and generate first information indicating the first activity parameter. As another example, the second device may obtain the first activity parameter from another device and generate first information indicating the first activity parameter. The aforementioned other device may refer to a device different from the first device and the second device.

[0124] This application does not limit the triggering conditions for the second device to generate the first information. As an example, the second device may generate the first information based on the request of the first device. For example, the second device may receive a request message from the first device, and the request message instructs the second device to send the first active parameter; the second device may generate the first information based on the request message. As another example, the second device may generate the first information based on the request of other devices. For example, the second device may receive a request message from other devices, and the request message instructs the second device to send the first active parameter to the first device; the second device may generate the first information based on the request message. As another example, the second device may actively generate the first information. That is, the second device may decide to generate the first information on its own, not based on the request of a certain device.

[0125] In some possible implementations, the method 400 further includes: (S440) the first device performs reasoning based on the N first modules according to the first activity parameter.

[0126] Optionally, the first activity parameter includes an activity parameter of an FNN module. The first device may set the element at the position of the activity parameter in the weight matrix of the FNN module to the activity parameter indicated by the first information, and set the elements at other positions in the weight matrix to 0, thereby obtaining a processed weight matrix. The first device may perform inference based on the input of the FNN module and the FNN module configured with the processed weight matrix.

[0127] It should be noted again that the above example uses one FNN module, but the present application does not limit the first activity parameter to include only the activity parameter of one FNN module. The first activity parameter may also include the activity parameters of more FNN modules.

[0128] Through the above embodiment, the first device used for inference may not store the full parameters of the first module, but obtain the active parameters in the full parameters of the first module through the first information. Compared with the solution of storing the full parameters locally in the first device and reading the active parameters from the full parameters, the above embodiment can save the storage space of the first device, so that the first device with limited storage space can run a large model with a large number of parameters, thereby improving the inference performance of the large model. For example, the first module can be an FNN module in a large model of a transformer structure. The ratio of the full parameters of the FNN module to the full parameters of the MHA module is approximately 2:1. If the full parameters of the FNN module are not stored in the end-side device, two-thirds of the flash memory space of the end-side device will be saved. The saved flash memory space can store the parameters of other modules, such as the parameters of the MHA module. In this way, the number of parameters of the large model can be increased by three times, thereby effectively improving the performance of the large model.

[0129] Optionally, the first activity parameter includes activity parameters of one or more MHA modules. Optionally, the first activity parameter includes activity parameters of one or more MHA modules and activity parameters of one or more FNN modules.

[0130] In some possible implementations, the method 400 further includes: (S410) the first device sends third information to the second device. Correspondingly, the second device receives the third information from the first device.

[0131] Optionally, the third information includes information indicating the position of the first activity parameter. The information indicating the position of the first activity parameter may indicate the position of the first activity parameter. As an example, the information indicating the position of the first activity parameter may include the specific position of the first activity parameter. For example, the position of the first activity parameter may be the position of an element in the weight matrix of the FNN module. For example, for a 2*4 weight matrix, the first row, second column and the second row, second column may be the position of the first activity parameter. As another example, the information indicating the position of the first activity parameter may include an identifier of the position of the first activity parameter. The first device may pre-acquire a correspondence between the positions of multiple activity parameters and multiple identifiers. For example, the first device may pre-acquire a mapping table that includes mappings between the positions of multiple activity parameters and multiple identifiers. Based on the identifier of the position of the first activity parameter, the first device may determine the position of the first activity parameter corresponding to the identifier. For example, identifier #1 corresponds to the first row, second column and the second row, second column. Thus, based on identifier #1 indicated by the information indicating the position of the first activity parameter, the first device may determine that the position of the first activity parameter is the first row, second column and the second row, second column.

[0132] For another example, the position of the first activity parameter may be the position of the active head of the MHA module. For example, for an MHA module having eight heads, the third and fourth heads among the eight heads may be active heads. In this way, the indication information of the position of the first activity parameter may indicate the third and fourth heads. The manner of indicating the third and fourth heads may be direct or indirect, and examples are not repeated here.

[0133] Optionally, the third information is used to request the first activity parameter. In other words, the third information can be used to request the second device to send the first activity parameter. Alternatively, the third information can be used to request the second device to send the first activity parameter to the first device.

[0134] Through the above embodiment, the first device can request to send the first activity parameter, and the request sent by the first device can carry information indicating the location of the first activity parameter, so that the value of the activity parameter indicated by the first information is accurate.

[0135] In some possible implementations, S430 includes: in response to the third information, the second device sends the first information to the first device.

[0136] In other words, the second device may send the first activity parameter to the first device based on the request of the first device.

[0137] It should be noted that this application does not limit the first device to reporting the indication information of the location of the first activity parameter to the second device. In other possible implementations, the second device can obtain the indication information of the location of the first activity parameter from other devices. The indication information of the location of the first activity parameter may be direct indication information, that is, the indication information includes location information; or the indication information of the location of the first activity parameter may be indirect indication information, and the second device can determine the location information of the first activity parameter based on the indication information.

[0138] In some implementations, the method 400 further includes: (S450) when the confidence level of the output value of the first model is greater than or equal to a first threshold, the first device sending second information to the second device. Correspondingly, the second device receives the second information from the first device.

[0139] Exemplarily, the confidence of the output value of the first model can be calculated based on the cosine similarity of the output values ​​of the two adjacent layers of the first model. However, this application does not limit the calculation method of the confidence of the output value of the first model, and the confidence of the output value of the first model can also be calculated in other ways. For example, a dedicated linear classifier can be trained based on a small AI model or an ML model, and the dedicated linear classifier can calculate its confidence using the output value of the first model. The first threshold can be preset. Exemplarily, the first threshold can take a larger value, for example, the first threshold can take 95%. This application does not limit the name of the first threshold, and the first threshold can also be called a confidence threshold, a preset threshold, a stop threshold or have other names.

[0140] Optionally, the second information is used to instruct to stop transmitting the first information. As an example, when the first device has requested the first activity parameter and the first device has not received the first information, S450 can be executed, that is, the first device can send the second information to the second device when the confidence condition is met, instructing the first device to no longer transmit the information indicating the first activity parameter. As another example, the first device has received part of the first information, for example, the first device has received information indicating the activity parameters of part of the first modules among N2 first modules. The first device can execute S450, that is, the first device can send the second information to the second device when the confidence condition is met, instructing the first device to stop transmitting the first information, that is, not to transmit the remaining part of the first information.

[0141] This application does not limit the name of the second information, and the second information may also be called stop information or have other names.

[0142] In some possible implementations, the N first modules include N1 first modules, the N1 first modules are modules of a first layer in at least one layer in the first model, and N1 is a positive integer less than or equal to N.

[0143] The first model may include at least one layer. For example, the first model may include at least one transformer layer. The N1 first modules may be modules of the first layer of the at least one layer. For example, if N1=1, the first module may be an FNN module in the first transformer layer of the first model. For another example, if N1=1, the first module may be an MHA module in the first transformer layer of the first model.

[0144] The N first modules include N1 first modules, and the first activity parameter may include the activity parameters of the N1 first modules in the first model. The N1 first modules may be modules in the first layer, and the first activity parameter may include the activity parameter of the first layer in the first model. In conjunction with S430, the second device may send the activity parameter of the first layer in the first model to the first device. In conjunction with S440, the first device may base reasoning on the activity parameter rather than the full parameter when performing reasoning based on the model in the first layer.

[0145] Through the above embodiment, the first device can perform reasoning based on the modules in the first layer of the first model based on the activity parameters indicated by the first information. Compared to the solution of reasoning based on the full parameters and the modules in the first layer, the embodiment of the present application does not need to load the full parameters of the modules in the first layer into the memory, thereby saving memory space of the first device and further improving the reasoning performance of the first model.

[0146] Figure 5 is a schematic flow chart of another communication method 500 provided in an embodiment of the present application. Method 500 can be combined with method 400. In method 500, the full parameters of the first of the multiple modules in the first layer of the first model can be stored locally in the first device; the full parameters of the second of the multiple modules in the first layer of the first model can be stored in the second device. That is, the first model needs to request the activity parameters of the second of the multiple modules in the first layer from the second device; the second device can determine the activity parameters (i.e., the first activity parameters) based on the full parameters of the second of the multiple modules in the first layer. Method 500 is introduced below in conjunction with Figure 5.

[0147] FIG5 assumes that the first module includes the second of multiple modules in a layer of the first model. It is assumed that the second module includes the first of multiple modules in a layer of the first module. However, it should be noted that the description of FIG5 is for example only and does not constitute a limitation of the present application. For example, the first module may include the first and second of multiple modules in a layer of the first model. In other words, in some other embodiments of the present application, the first module may include all modules in a layer of the first model.

[0148] S510: The first device predicts information about the position of the second activity parameter based on the first input value. It should be noted that in FIG5 , the operation of S510 is shown in dotted lines, indicating that S510 is an optional operation when combined with method 400 .

[0149] Optionally, the N1 first modules are the second of the multiple modules in the first layer of the first model, where N1 = 1. For example, the first model is a large model with a transformer structure. The first of the multiple modules in the first transformer layer of the first model is an MHA module, and the second of the multiple modules is an FNN module. In other words, the first module may be an FNN module in the first transformer layer of the large model.

[0150] Optionally, the second activity parameter in S510 is an activity parameter of the first of the multiple modules in the first layer of the first model. For example, the first module may be the second of the multiple modules in the first layer of the first model, and the second activity parameter may be the activity parameter of the previous module of the first module in the first layer of the first model. For another example, the first of the multiple modules in the first layer of the first model may be an MHA module. That is, the second activity parameter may be the activity parameter of the MHA module in the first layer of the first model.

[0151] This application does not limit the name of the second activity parameter. The second activity parameter may also be called a parameter, an activity head, an activity weight, an activity matrix, or have other names.

[0152] Optionally, the first input value is an input value of a first of a plurality of modules in a first layer of the first model. For example, the first input value may be an input value of an MHA module in a first transformer layer of the first model. In another example, the first input value may be an embedded vector or tensor of a user's request to the large model.

[0153] Exemplarily, the first device may use a prediction module to perform predictions. Specifically, the prediction module may predict information about the location of the second activity parameter based on the first input value. Optionally, the prediction module is deployed in the first device; in other words, the first device includes the prediction module. Exemplarily, the prediction module may be a second model. For example, the second model may be an AI model or an ML model. In another example, the second model may be a DNN model.

[0154] In some possible implementations, method 500 further includes: (S520) the first device determines the second activity parameter based on information about the location of the second activity parameter. For example, the first device may store the full parameters of the first of the multiple modules in the first layer of the first model (e.g., the MHA module). In this way, the first device may determine the value of the second activity parameter based on information about the location of the second activity parameter and the stored full parameters. In another possible implementation, the first device sends a request message to the second device; the second device receives the request message. The request message may be used to request the second activity parameter. In this case, the second activity parameter may be obtained from the second device. The second device may pre-store the full parameters of the first of the multiple modules in the first layer of the first model (e.g., the MHA module), thereby determining the requested activity parameter from the stored full parameters based on the request information of the first device.

[0155] In some possible implementations, method 500 further includes: (S530) the first device performs reasoning based on the first of the multiple modules in the first layer in the first model according to the second activity parameter. For example, the second activity parameter is the activity parameter of the MHA module in the first layer of the first model. The first device may update the weight matrix of the MHA module with the second activity parameter. Specifically, the weight submatrix of the inactive head in the MHA module is set to 0, and the weight submatrix of the active head is not affected. The first device may perform reasoning based on the second activity parameter (i.e., the non-zero weight submatrix of the updated MHA module, or the weight matrix of the updated MHA module) according to the first input value.

[0156] For example, method 500 can be applied to scenarios involving digital chat. For example, a user can send a text message, picture message, or voice message to a virtual digital agent on a mobile phone and receive a text message, picture message, or voice message in reply from the digital agent. In this scenario, the virtual digital agent can be driven by a first model. For example, a message sent by a user to the virtual digital agent can be input into a first model; after reasoning based on the input, the first model can understand and output a message for replying to the user. In this scenario, the data input to the first model may include information such as the user's chat history. Therefore, a large model can be deployed on an end-side device such as a mobile phone, and the data on the end-side device is not uploaded, thereby protecting the user's privacy. Furthermore, the storage space (e.g., memory or flash memory) of end-side devices such as mobile phones is limited, making it difficult to deploy large models with large parameter counts. Furthermore, when users chat with a virtual digital agent, they do not have high requirements for the delay in the virtual digital agent's reply message, that is, they do not require the virtual digital agent to reply to messages immediately.

[0157] Through the above embodiment, the first device can predict the activity parameters of the first of the multiple modules in the first layer of the first model, and thus perform reasoning based on the first of the multiple modules according to the activity parameters of the first of the multiple modules. Compared with the solution of performing reasoning based on the full parameters and the first of the multiple modules in the first layer, the embodiment of the present application does not need to load the full parameters of the first of the multiple modules in the first layer into the memory, thereby saving memory space of the first device, thereby further improving the reasoning performance of the first model.

[0158] S540: The first device predicts the position information of the first activity parameter according to the second input value. It should be noted that in FIG5 , the operation of S540 is shown in dotted lines, indicating that S540 is an optional operation when combined with method 400.

[0159] Optionally, the second input value may be an input value of a second one of the multiple modules in the first layer of the first model or an output value of a first one of the multiple modules in the first layer of the first model. For example, the second input value may be an output value of an MHA module in the first transformer layer of the first model. For another example, the second input value may be an input value of an FNN module in the first transformer layer of the first model. Exemplarily, the second one of the multiple modules in the first layer of the first model may be an FNN module. Exemplarily, the second one of the multiple modules in the first layer of the first model may be the aforementioned first module.

[0160] Exemplarily, the first device may use a prediction module to perform predictions. Specifically, the prediction module may predict information about the location of the first activity parameter based on the second input value. Optionally, the prediction module is deployed in the first device; in other words, the first device includes the prediction module. Exemplarily, the prediction module may be a second model. For example, the second model may be an AI model or an ML model. In another example, the second model may be a DNN model.

[0161] In some possible implementations, S410 is performed after S540. For example, the information indicating the position of the first activity parameter included in the third information may be predicted by the first device according to the second input value.

[0162] Through the above embodiment, the first device can predict the activity parameters of the second of the multiple modules in the first layer of the first module, and thus perform reasoning based on the second of the multiple modules according to the activity parameters of the second of the multiple modules. Compared with the solution of reasoning based on the full parameters and the second of the multiple modules in the first layer, the embodiment of the present application does not need to load the full parameters of the second of the multiple modules in the first layer into the memory, thereby saving memory space of the first device, thereby further improving the reasoning performance of the first model.

[0163] In some possible implementations, the N first modules also include N2 first modules, the first model includes N2 layers, each of the N2 layers includes one first module among the N2 first modules, N2≥2, and N2 is a positive integer less than or equal to N.

[0164] The N first modules may include N2 first modules. In this way, the first activity parameter may include the activity parameters of N2 first modules. N2 ≥ 2, that is, the first activity parameter may include the activity parameters of multiple first modules. The above-mentioned multiple first modules may be located in different layers respectively, and therefore, the first activity parameter may include the activity parameters of multiple layers. In order to reduce the transmission delay, the second device may send down the activity parameters of multiple layers at a time, reducing the number of times the activity parameters are sent down, thereby reducing the signaling overhead caused by the frequent sending of activity parameters. Optionally, N may be an integer from 1 to 8.

[0165] It should be noted that the first model may include N2 layers, but is not limited to N2 layers. For example, the first model may include more than N2 layers. Unlike other layers in the first model, the N2 layers in the first model may correspond one-to-one to the N2 first modules. For example, the m-th layer among the N2 layers of the first model may include the m-th first module among the N2 first modules, where m is an integer from 1 to N2. In this way, the first activity parameters issued by the second device may include the activity parameters of the N2 first modules in the N2 layers. The layers other than the N2 layers in the first model may include the first module or may not include the first module. In other words, the first activity parameters issued by the second device may include the activity parameters of the modules in the layers other than the N2 layers in the first model, or may not include the activity parameters of the modules in the layers other than the N2 layers in the first model.

[0166] The present application does not limit the position of the N2 layers in all the layers in the first model. For example, the N2 layers may be the first N2 layers of the first model. That is, the N2 layers may include the first layer to the N2th layer in the first model. For another example, the N2 layers may be the middle N2 layers of the first model. That is, the N2 layers may include the xth layer to the x+N2-1th layer in the first model, where x may be an integer greater than 1. In the above two examples, the N2 layers are numbered continuously in all the layers of the first model; in other words, no other layers are inserted in these N2 layers (that is, they may or may not include the layers of the first module). The above-mentioned first module can be understood as the module to which the activity parameters issued by the second device belong. However, the present application does not limit this, and the N2 layers may also be numbered discontinuously in all the layers of the first model; in other words, other layers may be inserted in these N2 layers.

[0167] Through the above embodiment, the second device can send activity parameters of multiple layers at one time, reducing the number of times the activity parameters are sent, thereby reducing the signaling overhead caused by frequent sending of activity parameters.

[0168] In conjunction with method 500 shown in FIG5 , the N first modules may include N1 first modules and N2 first modules. Thus, the first activity parameters may include activity parameters for the N1 first modules and the N2 first modules. It should be noted that in some possible implementations, the first information may be transmitted twice. In one transmission, the first information indicates a portion of the activity parameters included in the first activity parameters, for example, the activity parameters for the N1 first modules. In the other transmission, the first information indicates another portion of the activity parameters included in the first activity parameters, for example, the activity parameters for the N2 first modules. However, the above is merely an example and does not constitute a limitation of the present application. For example, in other possible implementations, the first information may be transmitted only once. The first activity parameters indicated by the first information may include activity parameters for the N1 first modules and the N2 first modules. For another example, in other possible implementations, the first information may be transmitted more times. The first activity parameters indicated by the first information in each transmission may include a smaller number of activity parameters than the first activity parameters indicated by the first information in each of the two transmissions.

[0169] Figure 6 is a schematic flow chart of another communication method 600 provided in an embodiment of the present application. Method 600 can be combined with method 400 and method 500. In method 600, the full parameters of the second module in the N2 layers of the first model can be stored locally in the first device; the full parameters of the first module in the N2 layers of the first model can be stored in the second device. That is, the first model needs to request the activity parameters of the first module in the N2 layers from the second device; the second device can determine the activity parameters (i.e., the first activity parameters) based on the full parameters of the first module in the N2 layers. Method 600 is introduced below in conjunction with Figure 6.

[0170] The solution shown in FIG6 assumes that the first module includes the second of the multiple modules in each of the multiple layers of the first model. It is assumed that the second module includes the first of the multiple modules in each of the multiple layers of the first module. However, it should be noted that the description of FIG6 is for example only and does not constitute a limitation of the present application. For example, the first module may include the first and second of the multiple modules in each of the multiple layers of the first model. In other words, in some other embodiments of the present application, the first module may include all the modules in the multiple layers of the first module.

[0171] At step S610, the first device predicts information about the position of the first activity parameter based on the input value of the layer immediately preceding the first layer in the N2 layers. It should be noted that in FIG6 , the operation of step S610 is shown as a dotted line, indicating that step S610 is optional when combined with methods 400 and 500.

[0172] It is understood that the first model includes other layers before the N2 layers. For example, the first model includes 16 layers, and the N2 layers are layers 2 to 9 of these 16 layers. Exemplarily, the prediction module can predict information about the location of the first activity parameter based on the input value of the layer immediately preceding the first layer in the N2 layers. This example is not repeated here; please refer to the description of methods 400 and 500 above.

[0173] The input value of the previous layer of the first layer among the N2 layers can be understood as the input value of the first of the multiple modules of the previous layer of the first layer among the N2 layers.

[0174] In some possible implementations, while S610 is being executed, the layer immediately preceding the first layer in the N2 layers can perform inference based on its input values. Thus, while the layer immediately preceding the first layer in the N2 layers is performing inference, the prediction module can predict information about the location of the first activity parameter, i.e., predict the activity parameters of the N2 first modules in the N2 layers following the layer being inferred. The layers for which the activity parameters are predicted are asynchronous with the layers where the model is being inferred, and thus the above scheme can also be referred to as asynchronous prediction.

[0175] Those skilled in the art will appreciate that due to residual connections, the output of each layer in a large model of a transformer structure changes very slowly. That is, the outputs of adjacent layers are highly similar, which makes the above-mentioned asynchronous prediction of multiple layers almost no reduction in the accuracy of predicting the location of activity parameters. In one possible implementation, N2 can be 8, that is, the first device predicts the location information of the activity parameters of the first module in the 8 layers below the layer based on the input of the layer.

[0176] Through the above embodiment, on the one hand, the first device can predict the information of the positions of the activity parameters of the multiple first modules respectively included in the multiple layers. Compared with the scheme of predicting the activity parameters of only one module at a time, the above embodiment can save computing resources. On the other hand, the first device predicts N2 layers based on the input value of the previous layer of the first layer in the N2 layers, rather than based on the input value of the first layer in the N2 layers. In this way, the first device can predict the information of the positions of the activity parameters of the first modules in the N2 layers when inferring the previous layer of the first layer in the N2 layers. That is to say, the layer for which the first device predicts the position of the activity parameters is not the layer being inferred, so the above embodiment will not affect the layer being inferred, that is, it will not reduce the inference performance of the large model.

[0177] In some other possible implementations, S610 can be replaced by: the first device predicts the information of the position of the first activity parameter based on the input value of the first layer among the N2 layers. Those skilled in the art can understand that the first device obtains the input value of the first layer among the N2 layers, indicating that the model reasoning is in the first layer among the N2 layers. At the same time, the prediction module is predicting the position of the first activity parameter, that is, predicting the activity parameters of the N2 first modules in the layer being reasoned and the N2-1 layers behind the layer. The layer for which the activity parameters are predicted is synchronized with the layer where the model reasoning is located, so the above scheme can also be called synchronous prediction. In another understanding, since the above scheme also predicts the layers behind the layer where the model reasoning is located, the above scheme can also be called asynchronous prediction.

[0178] In conjunction with method 400, in some possible implementations, S410 is performed after S610. For example, the position of the first activity parameter indicated by the indication information included in the third information may be predicted by the first device based on the input value of the layer above the first layer in the N2 layers (or, the input value of the first layer in the N2 layers). In conjunction with method 400, in some possible implementations, S430 may be performed after S610.

[0179] Optionally, each of the N2 layers further includes a second module, which is one of the N2 second modules in the first model. In other words, each of the N2 layers may include a first module and a second module. Exemplarily, the second module may be an MHA module.

[0180] In some implementations, the method 600 further includes: (S620) the first device predicts information about the position of the third activity parameter based on an input value of a previous layer of the first layer in the N2 layers.

[0181] Optionally, the third activity parameter is an activity parameter of the N2 second modules. For example, the third activity parameter is an activity parameter of the N2 MHA modules in the N2 layer. This application does not limit the name of the third activity parameter, and the third activity parameter may also be referred to as a parameter, an active neuron, an activity weight, an active element, or other names.

[0182] For example, the prediction module can predict the position information of the third activity parameter based on the input value of the previous layer of the first layer in the N2 layers.

[0183] In some possible implementations, S610 and S620 may be executed simultaneously, that is, the prediction module may predict information about the position of the first activity parameter and information about the position of the third activity parameter based on the input value of the previous layer of the first layer in the N2 layers.

[0184] In some possible implementations, while S620 is being executed, the layer above the first layer in the N2 layers can perform inference based on its input values. Thus, while the layer above the first layer in the N2 layers is performing inference, the prediction module can predict information about the location of the third activity parameter, i.e., predict the activity parameters of the N2 second modules in the N2 layers following the layer being inferred. The layers for which the activity parameters are predicted are asynchronous with the layers where the model is being inferred, and thus the above scheme can also be referred to as asynchronous prediction.

[0185] Through the above embodiment, on the one hand, the first device can predict the location information of the activity parameters of multiple second modules included in multiple layers. Compared with a solution that predicts the activity parameters of only one module at a time, this embodiment can save computing resources. On the other hand, the layer for which the first device predicts the location of the activity parameters is not the layer currently being inferred. Therefore, this embodiment does not affect the layer currently being inferred, that is, it does not reduce the inference performance of large models.

[0186] In some other possible implementations, S620 can be replaced by: the first device predicts the information of the position of the third activity parameter based on the input value of the first layer among the N2 layers. Those skilled in the art can understand that the first device obtains the input value of the first layer among the N2 layers, indicating that the model reasoning is in the first layer among the N2 layers. At the same time, the prediction module is predicting the position of the third activity parameter, that is, predicting the activity parameters of the N2 second modules in the layer being reasoned and the N2-1 layers behind the layer. The layer for which the activity parameters are predicted is synchronized with the layer where the model reasoning is located, so the above scheme can also be called synchronous prediction. In another understanding, since the above scheme also predicts the layers behind the layer where the model reasoning is located, the above scheme can also be called asynchronous prediction.

[0187] In some possible implementations, method 600 further includes: (S630) the first device determines the third activity parameter based on the information about the location of the third activity parameter. For example, the first device may store the full parameters of the N2 second modules (e.g., MHA modules) in the N2 layers of the first model. In this way, the first device may determine the value of the third activity parameter based on the information about the location of the third activity parameter and the stored full parameters. In another possible implementation, the first device sends a request message to the second device; the second device receives the request message. The request message may be used to request the third activity parameter. In this case, the third activity parameter may be obtained from the second device. The second device may pre-store the full parameters of the second modules (e.g., MHA modules) of the N2 layers of the first model, thereby determining the requested activity parameter from the stored full parameters based on the request information of the first device.

[0188] In some possible implementations, S630 may be executed before S610. For example, the execution order may be: S620, S630, S610. In other possible implementations, S630 may be executed after S610. For example, the execution order may be: S610, S620, S630. For another example, the execution order may be: S620, S610, S630. However, this application is not limited to this. For example, S630 may also be executed simultaneously with S610.

[0189] In some possible implementations, the method 600 further includes: (S640) the first device performs inference on the N2 layers according to the first activity parameter and the third activity parameter.

[0190] In some possible implementations, reasoning on the N2 layers includes: the first device uses the output value of the first module in the kth layer of the N2 layers as the input value of the second module in the kth layer. The above scheme can also be called sequential reasoning.

[0191] In some possible implementations, reasoning on the N2 layers includes: the first device determining a third input value based on an output value of a first module in a kth layer among the N2 layers and an output value of a second module in the kth layer, where k is a positive integer less than or equal to N2. This scheme may also be referred to as parallel reasoning.

[0192] For example, the first device may add the output value of the first module in the kth layer and the output value of the second module in the kth layer to obtain the third input value. For another example, the first device may add the output value of the first module in the kth layer and the output value of the second module in the kth layer, and multiply the added value by a non-zero coefficient to obtain the third input value.

[0193] Optionally, the third input value is used as input for a first module in a layer next to the k-th layer, and as input for a second module in a layer next to the k-th layer.

[0194] Figure 7 is a schematic diagram of another communication method provided by an embodiment of the present application. Figure 7 illustrates an embodiment combining synchronous prediction with asynchronous prediction. In short, synchronous prediction is used for the first layer of the large model, and asynchronous prediction (or synchronous prediction) is used for multiple layers after the first layer. It should be noted that the embodiment shown in Figure 7 is merely an example and does not constitute a limitation of the present application. The following is an introduction with reference to Figure 7.

[0195] Figure 7 illustrates the inference process for the first 2+2N2-1 layers of this large model. For ease of description, the first layer, the second to the 2+N2-1 layers, and the 2+N2 to 2+2N2-1 layers are represented by short-line rounded rectangles, respectively. The layer number of the module is indicated by a subscript. For example, the MHA module in the first layer is represented by MHA1. The remaining descriptions of the MHA module, FNN module, and prediction module are similar to those in Figure 3 and are not repeated here.

[0196] Specifically, after the user's request for the large model is embedded, the resulting vector or tensor can serve as the first "input" in Figure 7. This first input can be the input to the first-layer MHA module, as shown by MHA1 and the dotted arrow pointing to it in Figure 7. Based on MHA1's input, the prediction module can predict the location of MHA1's activity parameters, as indicated by P1 and the solid arrows associated with P1 in Figure 7. The first device can retrieve MHA1's activity parameters from its memory. The first device can perform inference based on MHA1 and its activity parameters to obtain MHA1's output. That is, when MHA1 performs inference, the first device can load a submatrix of some active heads into its memory. The output of MHA1 can serve as the input to the first-layer FNN module, as shown by FNN1 and the dotted arrow pointing to it in Figure 7. Based on FNN1's input, the prediction module can predict the location of FNN1's activity parameters, as indicated by P2 and the solid arrows associated with P2 in Figure 7. The first device can request the second device to transmit FNN1's activity parameters. The second device can store all parameters of the FNN module. For example, the second device can store the values ​​of all elements in a 4*4 weight matrix and the corresponding position of each element. The second device can determine the value of the activity parameter based on the position of the activity parameter. For example, if the activity parameter is located in the first row, third column, and the second row, fourth column, the second device can determine the value of the element in the first row, third column, and the value of the element in the second row, fourth column in the 4*4 weight matrix. The second device can transmit the activity parameter to the first device. For example, the second device can transmit first information to the first device, indicating the activity parameter of FNN1 (i.e., the N1 first modules). For another example, the second device can transmit information to the first device indicating the values ​​of the elements in the first row, third column, and the second row, fourth column. The process of the first device requesting the activity parameter and the second device transmitting the activity parameter is shown in the dotted double-headed arrow in Figure 7. The first device can perform inference based on FNN1 and its activity parameter; that is, when FNN1 performs inference, the first device can load some weight parameters into the memory of the first device.

[0197] Those skilled in the art will appreciate that the two inputs to the first layer of a large transformer-based model differ significantly. The above solution employs synchronous prediction for the first transformer layer, using the input to MHA1 to predict the location of MHA1's activity parameters, and the input to FNN1 to predict the location of FNN1's activity parameters. This solution conserves memory in the first device and provides more accurate predictions of the activity parameters for the first layer.

[0198] The output of FNN1 can be used as the input of the MHA module in the second layer, as shown by MHA2 and the dotted arrow pointing to MHA2 in Figure 7. The prediction module can predict the activity parameters of the MHA modules and FNN modules in the 2nd to 2+N2-1th layers based on the input of MHA2, as shown by P3 and the solid arrows related to P3 in Figure 7. For the MHA module, for example, MHA2 to MHA 2+N2-1 , the first device can obtain the activity parameters of the MHA module from the storage space of the first device. For the FNN module, for example, FNN2 to FNN 2+N2-1 , the first device can request the second device to send FNN2 to FNN 2+N2-1 The second device may send the activity parameter to the first device. For example, the second device may send a first message to the first device, and the first message is used to indicate the activity parameter of FNN2 to FNN 2+N2-1 (i.e., the activity parameters of the N2 first modules). The description of the MHA modules and FNN modules in the 2nd + N2th layer to the 2nd + 2N2-1th layer refers to the description of the 2nd layer to the 2nd + N2-1th layer above, and will not be repeated here.

[0199] Those skilled in the art will appreciate that the outputs of the second and subsequent layers of a large transformer-based model are highly similar, so the inputs from one layer can be used to predict the activity parameters of multiple subsequent layers at once. This allows the second device to transmit the activity parameters of multiple layers at once, effectively reducing the overhead associated with information transmission between the first and second devices.

[0200] Figure 8 is a schematic flow chart of another communication method 800 provided in an embodiment of the present application. In Figure 8, the first device may include a user operation module, a large model reasoning user interface (UI), a prediction module, and an end-side reasoning module. Optionally, the user operation module and the large model reasoning UI may also be devices used in conjunction with the first device. Method 800 can be combined with the process of the first device and the second device shown in Figure 7. It should be noted that method 800 is only an example and does not constitute a limitation of the present application. Method 800 is introduced below.

[0201] At step S802, the user operation module sends a request message to the large model inference user interface (UI). In other words, the user operation module may initiate a request to the large model inference UI. The user operation module may convert the user's operation into information recognizable by the first device. Optionally, the request message includes user input.

[0202] At step S804, the large model inference UI sends the user input to the prediction module, which then receives the user input. Before the prediction module performs prediction, the first device may perform embedding or other processing on the user input, and the resulting vector or tensor may serve as the input to the first layer MHA1.

[0203] S810 , the prediction module predicts the position of the activity parameter of MHA1 based on the input of MHA1.

[0204] S812 : The prediction module may send indication information of the location of the activity parameter of MHA1 to the end-side inference module, and the end-side inference module receives the indication information of the location of the activity parameter of MHA1 .

[0205] At step S814, the device-side inference module performs inference based on the activity parameters of MHA1, the input of MHA1, and MHA1 to obtain the output of MHA1. The device-side inference module may first obtain the activity parameters of MHA1 from the storage space of the first device based on the location of the activity parameters of MHA1, and then perform inference based on the activity parameters of MHA1, the input of MHA1, and MHA1.

[0206] S816 , the client-side inference module sends the output of MHA1 (ie, the input of FNN1 ) to the prediction module; the prediction module receives the output of MHA1 (ie, the input of FNN1 ).

[0207] S818, the prediction module predicts the position of the activity parameter of FNN1 based on the input of FNN1.

[0208] At step S820, the prediction module may send information requesting the activity parameters of FNN1 to the second device; the second device may receive the information requesting the activity parameters of FNN1. The information requesting the activity parameters of FNN1 may include information indicating the location of the activity parameters of FNN1. It should be noted that method 800 is provided for ease of understanding only; the operation of sending the request to the second device may also be performed by other modules, such as a sending module.

[0209] S822: The second device sends the activity parameters of FNN1 to the first device; the first device receives the activity parameters of FNN1. In some possible implementations, the client-side inference module in the first device may receive the activity parameters of FNN1. However, this application is not limited thereto, and the receiving operation of the first device may also be performed by other modules, such as a receiving module.

[0210] S824 : The client-side inference module performs inference based on the activity parameters of FNN1 , the input of FNN1 , and FNN1 to obtain the output of FNN1 .

[0211] S830 : The end-side inference module sends the output of FNN1 (ie, the input of MHA2 ) to the prediction module; the prediction module receives the output of FNN1 (ie, the input of MHA2 ).

[0212] S832, the prediction module predicts MHA2 to MHA based on the input of MHA2 2+N2-1 The location of the active parameters, and FNN2 to FNN 2+N2-1 The location of the activity parameter.

[0213] S834, the prediction module may send a request to the second device for FNN2 to FNN 2+N2-1 The second device may receive information for requesting FNN2 to FNN 2+N2-1 The activity parameter information of FNN2 is used to request FNN 2+N2-1 The activity parameter information can include FNN2 to FNN 2+N2-1 It should be noted that method 800 is only for ease of understanding, and the operation of sending a request to the second device can also be performed by other modules. For example, a sending module.

[0214] S836: The prediction module can send MHA2 to MHA to the end-side inference module. 2+N2-1 The terminal side inference module can receive MHA2 to MHA 2+N2-1 An indication of the location of the active parameter.

[0215] S838, the second device sends FNN2 to FNN to the first device 2+N2-1 In some possible implementations, the client-side inference module in the first device may receive the activity parameters from FNN2 to FNN 2+N2-1 However, this application is not limited to this, and the operation of receiving by the first device can also be performed by other modules. For example, a receiving module.

[0216] It should be noted that the execution order of S834 to S838 is not limited. For example, the execution order is as follows: S834, S838, S836. For another example, the execution order is as follows: S836, S834, S838.

[0217] S840: The client-side inference module performs inference based on the modules in the second layer to the 2+N2-1 layer, the module activity parameters, and the module input, and finally outputs the FNN. 2+N2-1 The output (ie MHA 2+N2 Input). Among them, for MHA2 to MHA 2+N2-1 The terminal side inference module can first calculate the value of MHA2 to MHA 2+N2-1 The location of the active parameters is obtained in the storage space of the first device, MHA2 to MHA respectively. 2+N2-1 activity parameters.

[0218] Subsequently, the prediction module can 2+N2 The input predicts the activity parameters of the modules in the subsequent N2 layers and performs similar operations as S832 to S840 until the inference result of the large model is output. The inference result of the large model can be returned to the large model inference UI.

[0219] In some possible implementations, the first model includes M layers, where M is a positive integer greater than N2, and the N2 layers are not numbered consecutively in the M layers. In other words, the first model includes M consecutively numbered layers, and the M layers include N2 layers.

[0220] The N2 layers are not numbered consecutively within the M layers. This means that within the N2 layers, there are two adjacent layers with discontinuous numbers. For example, the M layers are the first 10 layers in the first model, and the N2 layers are the 2nd, 3rd, 5th, and 6th layers. The 3rd layer is adjacent to the 5th layer, but the numbers 3 and 5 are not consecutive.

[0221] Those skilled in the art will understand that randomly skipping some transformer layers, such as 25% of all the layers included in the large model, will not significantly reduce the performance of the large model, and can reduce the delay in reasoning. The large model in the embodiment of the present application supports random skipping of one layer of reasoning. For example, one transformer layer can be skipped for every 4 transformer layers. Exemplarily, the N2 layers are the 2nd layer, the 3rd layer, the 5th layer and the 6th layer, respectively. The output of the 2nd layer can be used as the input of the 3rd layer, and the output of the 3rd layer can be used as the input of the 5th layer; that is, the 4th layer is skipped and does not participate in reasoning. For another example, N2 is 8, and the large model can skip the 4th and 8th layers during the reasoning process.

[0222] Accordingly, the solution provided in the embodiment of the present application can skip some layers when predicting multiple layers following the currently inferred layer, that is, the activity parameters of these layers are not predicted. In conjunction with the description of S410, the first information sent by the second device can indicate the first activity parameter, that is, the activity parameters of N first modules. The N first modules can include N2 first modules, and the N2 first modules belong to N2 layers respectively. In this way, the first activity parameter can include the activity parameters of the first modules in the N2 layers. The numbering of the N2 layers in the M layers is discontinuous, indicating that the activity parameters indicated by the second device do not belong to N2 consecutively numbered layers, but to N2 discontinuously numbered layers, that is, the activity parameters indicated by the second device skip one or more layers. In conjunction with the description of S610, the first activity parameters predicted by the first device are not the activity parameters of N2 consecutively numbered layers, but the activity parameters of N2 discontinuously numbered layers, that is, the first device skips some layers when predicting the activity parameters, thereby reducing the prediction delay.

[0223] Through the above embodiment, the first activity parameter indicated by the first information may include activity parameters of discontinuously numbered layers, that is, these activity parameters may skip one or more layers, thereby reducing the amount of information transmitted in the first information and lowering the delay.

[0224] In some possible implementations, N2 is related to resources and / or inference latency requirements for transmitting the first information.

[0225] Optionally, the resources used to transmit the first information may include bandwidth, spectrum width, time slot, multiple antenna beams, etc. For example, the larger the bandwidth used to transmit the first information, the larger N2 may be. Conversely, the smaller the bandwidth used to transmit the first information, the smaller N2 may be.

[0226] The inference latency requirement may refer to the latency requirement for inference performed by the first device. For example, the first device may need to complete inference on one layer within 3 seconds. The higher the inference latency requirement, or the shorter the required inference latency, the larger N2. Conversely, the lower the inference latency requirement, or the longer the required inference latency, the smaller N2 can be.

[0227] Optionally, the inference latency requirement may include an upper limit on the latency for the first device to obtain the first information. For example, the first device needs to complete the inference of a layer within 3 seconds, and the time spent by the first device on inference based on the activity parameter is 2 seconds. Then, the upper limit on the latency for the first device to obtain the first information indicating the activity parameters of N2 first modules (including the layer and the N2-1 layers following the layer) is 1 second. It is required that the latency for the first device to obtain the first information cannot exceed 1 second, and correspondingly, the latency for the second device to send the first information cannot exceed 1 second; in other words, the first device needs to obtain the first information within 1 second, thereby obtaining the activity parameters of the N2 first modules.

[0228] Taking GPT3 as an example, GPT3 has a total of 96 transformer layers. The storage space occupied by the active parameters of each transformer layer is 175B×0.15 / 96×4=0.09375GB=93.75MB. Among them, the storage space occupied by the active parameters of the FNN module is 93.75×2 / 3=62.5MB. If the bandwidth is 200 megabytes per second (MB / s) and the upper limit of the latency for the first device to receive the first information is 1s, then N2 can be 3. In this way, the transmission of the first information can meet the inference latency requirements within the given resources.

[0229] Through the above embodiment, the value of N2 can be related to the resources used to transmit the first information and / or the inference latency requirement, so that the transmission of the first information can meet the resource constraints and / or inference latency requirements. The above solution can ensure that the first information can be transmitted smoothly, or can ensure that the transmission of the first information can meet the inference latency requirements of the first model, thereby further ensuring the inference performance of the first model.

[0230] Figure 9 is a schematic diagram of a large model for parallel reasoning provided in an embodiment of the present application. It should be noted that Figure 9 is only an example and does not constitute a limitation of the present application.

[0231] Referring to Figure 9 , the input of the first layer can serve as the input of MHA1 and FNN1. Based on the outputs of MHA2 and FNN2, the output of the first layer can be determined, serving as the input of the second layer. The reasoning method shown in Figure 9 can be referred to as parallel reasoning. It should be noted that in S650 , k is a positive integer less than or equal to N2. In some possible implementations, k is a positive integer ranging from 1 to N2, meaning that each of the N2 layers utilizes the aforementioned parallel reasoning method. In other possible implementations, k is a partial positive integer ranging from 1 to N2, meaning that a portion of the N2 layers (or one or more layers) utilize the aforementioned parallel reasoning method, while the remaining layers continue to utilize a reasoning method similar to that in Figures 3 and 7 . This reasoning method can be referred to as sequential reasoning, meaning that the input of a layer serves as the input of the MHA module of that layer, the output of the MHA module serves as the input of the FNN module of that layer, and the output of the FNN module serves as the output of that layer.

[0232] Compared to sequential reasoning, parallel reasoning can reduce the number of inputs per layer by half. For example, if N2 is 10, if all 10 layers use sequential reasoning, each layer has 2 inputs, for a total of 20 inputs. However, if all 10 layers use parallel reasoning, each layer has only one input, for a total of 10 inputs.

[0233] Since the number of inputs per layer is reduced by half, and the rate of change between adjacent inputs (for example, the inputs of adjacent modules in sequential reasoning, or the inputs of adjacent layers in parallel reasoning) is constant, the use of parallel reasoning allows the prediction module to predict the positions of activity parameters for twice as many layers as compared to the use of sequential reasoning. For example, if N2 is 8 when using sequential reasoning, that is, when predicting the positions of activity parameters of 8 layers at a time, the predicted positions of activity parameters will be more accurate. Then, when using parallel reasoning, N2 can be 16, that is, the positions of activity parameters of 16 layers can be predicted at a time (assuming that these 16 layers all use parallel reasoning), and the predicted positions of activity parameters of 8 layers can be as accurate as those of the 8 layers predicted when using sequential reasoning.

[0234] Through the above embodiment, one or more layers in the N2 layers can adopt the form of parallel reasoning, that is, when the first module of a layer is performing reasoning, the second module of this layer can also perform reasoning. Sequential reasoning is to use the output obtained after the second module of a layer completes reasoning as the input of the second module of this layer. Compared with the layer using sequential reasoning, the number of inputs of the layer using parallel reasoning can be reduced from 2 to 1. The number of inputs is reduced, and the rate of change of adjacent inputs is constant, which can increase the value of N2 that can be selected, so that the activity parameters indicated by the first information sent by the second device can be the activity parameters of more layers. In other words, the second device can send the activity parameters of more layers at one time, thereby further reducing the transmission delay from the second device to the first device.

[0235] In conjunction with some modules of Figure 8, in some possible implementations, the end-side reasoning module can determine the confidence of the output value of the first model, and when the confidence of the output value of the first model is greater than or equal to a first threshold, send an instruction to the prediction module, which is used to instruct the prediction module to stop prediction. In some possible implementations, when the confidence of the output value of the first model is greater than or equal to the first threshold, the first model can stop reasoning. In some possible implementations, when the confidence of the output value of the first model is greater than or equal to the first threshold, the first model can send the output value of the first model (or the reasoning result of the first model) to the large model reasoning UI and / or the user operation module.

[0236] Through the above embodiment, when the confidence level of the output value of the first model meets the conditions, the first device can send the second information, and the second information can indicate to stop transmitting the first information, thereby reducing the waste of resources caused by transmitting the first information. In addition, the above embodiment can be applied to the scenario of early exiting of reasoning. In the scenario where the first model exits reasoning early, the exit of the first model from reasoning is not due to the completion of reasoning for all layers of the first model, but because the confidence level of the output value of the first model meets the conditions. Early exit from reasoning can prevent some layers in the first model from participating in reasoning, thereby reducing the reasoning delay. The above embodiment can avoid the second device from transmitting information indicating active parameters used for reasoning when the first model exits reasoning early, thereby saving transmission resources.

[0237] The following is an introduction to the device embodiment corresponding to the method embodiment of the present application. The following is only a brief introduction to the device, and the specific implementation steps and details of the solution can be referred to the method embodiment above.

[0238] To implement the various functions of the method provided herein, the communication device may include hardware structures and / or software modules, and the aforementioned functions may be implemented in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular one of the aforementioned functions is implemented in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0239] Figure 10 is a schematic block diagram of a communication device 1000 according to an embodiment of the present application. The communication device 1000 includes a processor 1010 and a communication interface 1020. Optionally, the processor 1010 and the communication interface 1020 may be interconnected via a bus. The communication device 1000 may be a first device or a second device. For example, the first device may be a terminal device or an end-side device; the second device may be a network device, a core network device, a cloud-side device, or an edge-side device.

[0240] Optionally, the communication device 1000 may further include a memory 1040. The memory 1040 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, erasable programmable read-only memory (EPROM), synchronous dynamic random access memory (SDRAM), hard disk drive (HDD), solid-state drive (SSD), or portable compact disc read-only memory (CD-ROM). The memory 1040 is used to store relevant instructions and / or data. The memory 1040 can be integrated with the processor 1010 or set separately.

[0241] For example, communication device 1000 may be a first device. The memory of the first device may include RAM, cache, SDRAM, or CD-ROM. Optionally, the memory of the first device may be used to load the first active parameters. The flash memory of the first device may include ROM, EPROM, HDD, or SSD. Optionally, the flash memory of the first device may store all parameters of the second module.

[0242] For example, the communication device 1000 may be the second device. The flash memory of the second device may include ROM, EPROM, HDD, or SSD. Alternatively, the flash memory of the first device may store all parameters of the first module.

[0243] The processor 1010 may be one or more central processing units (CPUs). In the case where the processor 1010 is a CPU, the CPU may be a single-core CPU or a multi-core CPU. However, this application is not limited thereto, and the processor 1010 may also be one or more GPUs, or one or more TPUs. The processor 1010 may be a signal processor, a chip, or other integrated circuit that can implement the method of the present application, or a portion of the circuit for processing functions in the aforementioned processor, chip, or integrated circuit. In addition, the communication interface 1020 may also be an input / output interface, which is used for input or output of signals or data, or may be an input / output circuit.

[0244] Exemplarily, the communication device 1000 is a first device, and the processor 1010 is used to perform the following operations: receive first information, which is used to indicate a first activity parameter, and the first activity parameter is an activity parameter of N first modules in the first model, where N is a positive integer; and perform reasoning based on the N first modules according to the first activity parameter.

[0245] Exemplarily, the communication device 1000 is a second device, and the processor 1010 is used to perform the following operations: generate first information, the first information is used to indicate a first activity parameter, the first activity parameter is the activity parameter of N first modules in the first model, the first activity parameter is used to perform inference based on the N first modules, N is a positive integer; send the first information.

[0246] The above content is only for exemplary description. The communication device 1000 is responsible for executing the methods or steps related to the first device or the second device in the above method embodiments.

[0247] In one possible implementation, the communication interface 1020 may be a transceiver. The transceiver may include a transmitter and a receiver, where the transmitter is configured to perform a sending operation and the receiver is configured to perform a receiving operation. For example, the processor 1010 is configured to control the transceiver to receive and / or send signals.

[0248] In a possible implementation, the communication interface 1020 may also be a communication circuit, a pin, an input / output interface, a bus, etc.

[0249] It should be noted that the communication device 1000 may include a transmitter but not a receiver. Alternatively, the communication device 1000 may include a receiver but not a transmitter. The specific implementation depends on whether the above solution executed by the communication device 1000 includes a sending action and a receiving action.

[0250] The above description is merely exemplary. For details, please refer to the contents of the above method embodiments. The implementation of each operation in FIG10 may also correspond to the corresponding description of the method embodiments shown in FIG4 to FIG9.

[0251] For example, the communication device 1000 may be used to implement the solutions shown in FIG. 4 to FIG. 9 .

[0252] Exemplarily, the communication device 1000 is a first device, and the communication interface 1020 can be used to receive first information.

[0253] Exemplarily, the communication device 1000 is a second device, and the communication interface 1020 can be used to send the first information.

[0254] For other implementations, please refer to the detailed description of the embodiments shown in Figures 4 to 9 above, which will not be repeated here. It should be understood that the specific process of each component performing the above corresponding process has been detailed in the above method embodiment, and for the sake of brevity, it will not be repeated here.

[0255] Figure 11 is a schematic block diagram of another communication device 1100 according to an embodiment of the present application. Communication device 1100 may be the first device or the second device, or a chip or module within the first device or the second device, and is configured to implement the methods described in the embodiments of Figures 4 to 9. For details, please refer to the relevant descriptions of the aforementioned method embodiments.

[0256] The communication device 1100 includes a transceiver unit 1110. The transceiver unit 1110 is described below by way of example.

[0257] The transceiver unit 1110 may include a transmitting unit and a receiving unit. The transmitting unit is used to perform the transmitting operation of the communication device, and the receiving unit is used to perform the receiving operation of the communication device. For ease of description, the embodiments of the present application combine the transmitting unit and the receiving unit into a single transceiver unit. This is described here for a unified explanation and will not be repeated later. The transceiver unit 1110 can implement corresponding communication functions. The transceiver unit 1110 can also be referred to as a communication interface or a communication module.

[0258] It should be noted that the communication device 1100 may include a sending unit but not a receiving unit. Alternatively, the communication device 1100 may include a receiving unit but not a sending unit. The specific implementation depends on whether the above solution executed by the communication device 1100 includes a sending action and a receiving action.

[0259] Exemplarily, the transceiver unit 1110 is used to receive first information and the like.

[0260] Optionally, the communication device 1100 may further include a processing unit 1120, which is configured to execute the contents of steps involving processing, coordination, etc. of the communication device 1100. Exemplarily, the processing unit 1120 is configured to perform reasoning based on the N first modules according to the first activity parameter.

[0261] Exemplarily, the transceiver unit 1110 is used to send first information, etc.

[0262] Optionally, the communication device 1100 may further include a processing unit 1120, which is configured to execute the contents of steps involving processing, coordination, etc. of the communication device 1100. Exemplarily, the processing unit 1120 is configured to generate the first information.

[0263] The above contents are merely exemplary descriptions, and the communication device 1100 is responsible for executing the relevant methods or steps in the above method embodiments.

[0264] Optionally, the communication device 1100 further includes a storage unit 1130, which is configured to store a program or code for executing the aforementioned method. Alternatively, the storage unit 1130 may be configured to store instructions and / or data, and the processing unit 1120 may read the instructions and / or data in the storage unit 1130 to enable the communication device 1100 to implement the aforementioned method embodiments. For example, the communication device 1100 may be configured to execute the solutions illustrated in Figures 4 to 9.

[0265] Exemplarily, the transceiver unit 1110 can be used to receive first information, where the first information is used to indicate a first activity parameter, where the first activity parameter is an activity parameter of N first modules in the first model, where N is a positive integer; the processing unit 1120 can be used to perform inference based on the N first modules according to the first activity parameter.

[0266] Exemplarily, the processing unit 1120 can be used to generate first information, which is used to indicate a first activity parameter, which is an activity parameter of N first modules in the first model, and the first activity parameter is used to perform inference based on the N first modules, where N is a positive integer; the transceiver unit 1110 can be used to send the first information.

[0267] For other implementations, please refer to the detailed description of the embodiments shown in Figures 4 to 9 above, which will not be repeated here. It should be understood that the specific process of each component performing the above corresponding process has been detailed in the above method embodiment, and for the sake of brevity, it will not be repeated here.

[0268] When the communication device 1100 in FIG10 is a chip, the communication interface 1120 may be a transceiver, input / output circuit, or communication interface of the chip. The processor 1110 may be a processor, microprocessor, or integrated circuit integrated on the chip. The sending operation of the first device or the second device in the above method embodiment may be understood as an output of the chip, and the receiving operation of the first device or the second device in the above method embodiment may be understood as an input of the chip.

[0269] When the communication device 1100 in FIG10 is a chip, the transceiver unit 1110 may be a transceiver, input / output circuit, or communication interface of the chip. The processing unit 1120 may be a processor, microprocessor, or integrated circuit integrated on the chip. The sending operation of the first device or the second device in the above method embodiment can be understood as the output of the chip, and the receiving operation of the first device or the second device in the above method embodiment can be understood as the input of the chip.

[0270] The present application also provides a chip, including a processor, for calling and executing instructions stored in a memory, so that a communication device equipped with the chip executes the methods in the above examples.

[0271] The present application also provides another chip, comprising: an input interface, an output interface, and a processor, wherein the input interface, the output interface, and the processor are connected via an internal connection path, and the processor is configured to execute code in a memory. When the code is executed, the processor is configured to execute the methods in the above examples. Optionally, the chip also includes a memory, which is configured to store computer programs or code.

[0272] The present application also provides a processor for coupling with a memory, and for executing the methods and functions involving a communication device or an encoding device in any of the above embodiments.

[0273] In another embodiment of the present application, a computer program product including a computer program or instructions is provided. When the computer program product is run on a computer, the method of the aforementioned embodiment is implemented.

[0274] The present application also provides a computer program. When the computer program is executed in a computer, the method of the aforementioned embodiment is implemented.

[0275] In another embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a computer, the method described in the above embodiment is implemented.

[0276] The present application also provides a communication system, which includes a first device and a second device. The first device and the second device are respectively used to execute the methods executed by the first device and the second device in the above embodiments.

[0277] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0278] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0279] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0280] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0281] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0282] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0283] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A communication method, characterized in that: The method comprises: receiving first information indicating a first activity parameter, where the first activity parameter is an activity parameter of N first modules in a first model, where N is a positive integer; According to the first activity parameter, reasoning is performed based on the N first modules.

2. The method according to claim 1, characterized in that The N first modules include N1 first modules, and the N1 first modules are modules of a first layer in at least one layer in the first model, where N1 is a positive integer less than or equal to N.

3. The method according to claim 2, characterized in that The N1 first modules are second among the multiple modules in the first layer of the first model, N1=1, and the method further includes: Predict information about the position of the first activity parameter based on a second input value, wherein the second input value is an input value of a second one of multiple modules in the first layer of the first model or an output value of a first one of multiple modules in the first layer of the first model.

4. The method according to claim 2 or 3, characterized in that The method further comprises: Based on the first input value, predict information about the position of a second activity parameter, where the second activity parameter is the activity parameter of the first of multiple modules in the first layer of the first model, and the first input value is the input value of the first of multiple modules in the first layer of the first model.

5. The method according to any one of claims 1 to 4, characterized in that The N first modules also include N2 first modules, the first model includes N2 layers, each of the N2 layers includes one first module among the N2 first modules, N2≥2, and N2 is a positive integer less than or equal to N.

6. The method according to claim 5, characterized in that The first model includes M layers, where M is a positive integer greater than N2, and the N2 layers are not numbered consecutively in the M layers.

7. The method according to claim 5 or 6, characterized in that N2 is related to the resource and / or inferred latency requirements for transmitting the first information.

8. The method according to any one of claims 5 to 7, characterized in that The method further comprises: Information about the position of the first activity parameter is predicted based on an input value of a layer above the first layer in the N2 layers.

9. The method according to any one of claims 5 to 8, characterized in that Each of the N2 layers further includes a second module, where the second module is one of the N2 second modules in the first model, and the method further includes: According to the input value of the previous layer of the first layer in the N2 layers, information about the position of the third activity parameter is predicted, and the third activity parameter is the activity parameter of the N2 second modules.

10. The method according to claim 9, characterized in that The second module is the multi-head attention MHA module.

11. The method according to any one of claims 5 to 10, characterized in that Each of the N2 layers further includes a second module, and the method further includes: A third input value is determined based on the output value of the first module in the kth layer among the N2 layers and the output value of the second module in the kth layer, and the third input value is used for the input of the first module in the next layer of the kth layer and the input of the second module in the next layer of the kth layer, where k is a positive integer less than or equal to N2.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: When the confidence level of the output value of the first model is greater than or equal to a first threshold, second information is sent, where the second information is used to instruct to stop transmitting the first information.

13. The method according to any one of claims 1 to 12, characterized in that The method further comprises: Third information is sent, where the third information includes information indicating a location of the first activity parameter, and the third information is used to request the first activity parameter.

14. The method according to any one of claims 1 to 13, characterized in that The first module is a feedforward neural network FNN module.

15. A communication method, characterized in that: The method comprises: Generate first information, where the first information is used to indicate a first activity parameter, where the first activity parameter is an activity parameter of N first modules in the first model, and the first activity parameter is used to perform inference based on the N first modules, where N is a positive integer; The first information is sent.

16. The method according to claim 15, characterized in that The N first modules include N1 first modules, and the N1 first modules are modules of a first layer in at least one layer in the first model, where N1 is a positive integer less than or equal to N.

17. The method according to claim 15 or 16, characterized in that The N first modules also include N2 first modules, the first model includes N2 layers, each of the N2 layers includes one first module among the N2 first modules, N2≥2, and N2 is a positive integer less than or equal to N.

18. The method according to claim 17, characterized in that The first model includes M layers, where M is a positive integer greater than N2, and the N2 layers are not numbered consecutively in the M layers.

19. The method according to claim 17 or 18, characterized in that N2 is related to the resource and / or inferred latency requirements for transmitting the first information.

20. The method according to any one of claims 15 to 19, characterized in that The method further comprises: Second information is received, where the second information is used to instruct to stop transmitting the first information.

21. The method according to any one of claims 15 to 20, characterized in that The method further comprises: Third information is received, the third information including information indicating a location of the first activity parameter.

22. The method according to claim 21, characterized in that The sending of the first information includes: In response to the third information, the first information is sent.

23. The method according to any one of claims 15 to 22, characterized in that The first module is a feedforward neural network FNN module.

24. A communication device, characterized in that: The method comprises at least one module or at least one unit, wherein the at least one module or the at least one unit is used to execute the method according to any one of claims 1 to 15, or the at least one module or the at least one unit is used to execute the method according to any one of claims 16 to 23.

25. A communication device, characterized in that: include: A processor, wherein the processor is configured to, by executing a computer program or instruction, cause the communication device to perform the method according to any one of claims 1 to 14, or cause the communication device to perform the method according to any one of claims 15 to 23.

26. The communication device according to claim 25, characterized in that The communication device further comprises a memory configured to store the computer program or the instructions.

27. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program or instructions, which, when executed on a computer, causes the method according to any one of claims 1 to 14 to be executed, or causes the method according to any one of claims 15 to 23 to be executed.

28. A computer program product, characterized in that The method comprises a computer program or an instruction. When the computer program or the instruction is executed, the method according to any one of claims 1 to 14 is implemented, or the method according to any one of claims 15 to 23 is implemented.

29. A communication system, characterized in that: The method comprises a first device and a second device, wherein the first device is used to execute the method according to any one of claims 1 to 14, and the second device is used to execute the method according to any one of claims 15 to 23.

30. A chip, characterized in that: The chip comprises a processor for calling a computer program or computer instruction in a memory so as to enable the chip to execute the method according to any one of claims 1 to 14; or, execute the method according to any one of claims 15 to 23.

Images