A model performance optimization method and related device

Abstract

The application discloses a model performance optimization method, which is applied to a terminal. In the method, the terminal sends a first request to a server; and within a first time, feedback information sent by the server for the first request is received, the feedback information being capable of being used to determine a first operation mode of each operator in a plurality of operators of a neural network model. Through the method, the optimal operation mode of each operator in the plurality of operators of the neural network model of the terminal can be obtained based on end-cloud interaction, so that the terminal can obtain better running performance when performing operation operation on the neural network model in the optimal operation mode, and the developer does not need to perform complex performance optimization operation on various different types and models of devices that the neural network model may be deployed on, thereby improving the optimization efficiency of the neural network model on the terminal.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a model performance optimization method and related equipment. Background Technology

[0002] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. Research in the field of AI includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0003] With the rapid development of AI, neural network models (such as deep neural network models) are deployed in various devices to play a role in the processing and analysis of various types of information, including images, videos, and audio. However, the composition of computing units differs across devices, and these units are better suited for different computational scenarios, leading to significant performance differences for the same neural network model running on different devices or computing units. Furthermore, the frequent updates and iterations of hardware and software within devices also affect the performance of neural network models on those devices.

[0004] Currently, in order to achieve better performance of neural network models on multiple devices, developers need to deploy the trained neural network models on multiple devices and perform targeted testing and optimization on each device. This optimization process is inefficient and involves a large workload. Summary of the Invention

[0005] This application provides a model performance optimization method to address the problem that current methods require developers to deploy trained neural network models on multiple devices and perform targeted testing and optimization on each device separately in order to achieve optimal performance of neural network models across various devices, resulting in low optimization efficiency. This application also provides corresponding apparatus, devices, computer-readable storage media, and computer program products.

[0006] The first aspect of this application provides a model performance optimization method applied to a terminal. The method includes: sending a first request to a server, the first request carrying operator information of a neural network model and operating environment information of the neural network model; and receiving feedback information from the server in response to the first request within a first time period, the feedback information being able to determine the first operation mode of each of the multiple operators of the neural network model.

[0007] In the first aspect, the first operation mode is used to indicate the setting method of the operating parameters when the operator performs operations at the terminal. For example, the operating parameters may include the operating frequency of the computing unit corresponding to the operator.

[0008] The feedback information can be used to determine the first operation mode of each operator among multiple operators in a neural network model. Specifically, the feedback information can be used to determine the first computational unit corresponding to each operator in the neural network model, and the first operation mode of each operator in the neural network model in the corresponding first computational unit. In this case, the feedback information can include any one of the following two situations: (1) The feedback information can include information on the model operation strategy corresponding to the neural network model, and the model operation strategy includes the first operation mode corresponding to each operator among multiple operators in the neural network model. (2) The feedback information includes the operator performance information corresponding to the neural network model, and the operator performance information included in the feedback information satisfies the specified optimization conditions. In this case, the operator performance information included in the feedback information satisfying the specified optimization conditions can be understood as the feedback information including sufficient operator performance information. In one example, determining that the operator performance information included in the feedback information meets the specified optimization conditions can be that the operator performance information included in the feedback information is related to each computing unit of the terminal, thereby enabling the terminal to determine from the computing units of the terminal that the computing unit with better performance for the corresponding operator is the first computing unit of the corresponding operator, and to determine the operation mode with better performance for the corresponding operator is the first operation mode of the corresponding operator.

[0009] As can be seen from the above, in this first aspect, for the neural network model of the terminal, the optimal operation mode (i.e., the first operation mode) of each operator among the multiple operators of the neural network model can be obtained based on the relevant feedback information of the server. This enables the terminal to achieve better running performance when performing operation on the neural network model in the corresponding first operation mode, without requiring developers to perform complex performance optimization operations for various types and models of devices that the neural network model may be deployed on, thereby improving the optimization efficiency of the neural network model on the terminal.

[0010] In one possible implementation of the first aspect, the method further includes: obtaining a first computational result of the data to be processed based on feedback information.

[0011] In this possible implementation, the data to be processed can come from the application corresponding to the neural network model. For example, the application could implement facial recognition, in which case the data to be processed in the application would be an image. Alternatively, the application could implement text translation, in which case the data to be processed in the application would be text information.

[0012] In one possible implementation of the first aspect, the first operation mode of each of the plurality of operators includes the operation mode of the corresponding operator in the corresponding first computing unit, wherein the first computing unit is one or more computing units of the terminal.

[0013] In this possible implementation, the type of computing unit may include a central processing unit (CPU), or one or more of a graphics processing unit (GPU), neural network processing unit (NPU), and tensor processing unit (TPU). The first computing unit may be one or more of the terminal's computing units. When there are multiple first computing units, the types of each first computing unit may be the same or different, and the hardware configurations of each first computing unit may be the same or different.

[0014] In one possible implementation of the first aspect, the feedback information also carries first reference information; the method further includes: after obtaining the first operation result, when the difference between the first performance information and the first reference information meets a preset condition, sending the first performance information to the server, wherein the first performance information is used to indicate the running performance of at least one operator with respect to the corresponding first operation method.

[0015] In this possible implementation, a difference between the first performance information and the first reference information meeting a preset condition can indicate a large difference between the two. When the operator performance information corresponding to the terminal during actual operation differs significantly from the operator performance information in the relation list, the server can update the corresponding operator performance information stored in the server based on the operator performance information actually used by the terminal during operation.

[0016] In one possible implementation of the first aspect, the method further includes: if no feedback information is received within a first time period, or if the feedback information cannot be used to determine the first operation method of each of the multiple operators of the neural network model, then based on the second operation method of each of the multiple operators of the preset neural network model, a second operation result of the data to be processed is obtained.

[0017] In this possible implementation, the feedback information sent by the server cannot be used to determine the first operation method of each of the multiple operators in the neural network model, because the feedback information cannot support the terminal in determining the first operation method corresponding to all operators in the neural network model.

[0018] For a certain operator in a neural network model, the feedback information cannot be used to determine the first operation mode corresponding to the operator. This may be because the feedback information does not contain information about the operator, or the operator performance information contained in the feedback information is incomplete. For example, it may lack performance information about certain computing units and / or certain operation modes in the terminal.

[0019] In this possible implementation, when the terminal fails to obtain an optimized operating strategy for the neural network model from the server, it can determine the second operation method corresponding to each of the multiple operators in the neural network model based on the model operating strategy already stored in the terminal, thereby optimizing the operating performance of the neural network model. In other words, in this possible implementation, the second operation method corresponding to each of the multiple operators in the neural network model can be determined by the terminal based on the operator information and operating environment information of the neural network model, as well as the model operating strategy stored locally in the terminal.

[0020] In this possible implementation, when the neural network model is loaded again, the operation of sending the first request to the server and subsequent operations can be re-executed to try to optimize the performance of the neural network model again.

[0021] In one possible implementation of the first aspect, the method further includes: after obtaining the second operation result, sending second performance information to the server, the second performance information being used to indicate the running performance of at least one operator with respect to the corresponding second operation method.

[0022] In some examples of this possible implementation, the terminal fails to obtain feedback information from the server regarding the first operation method corresponding to each of the multiple operators used to determine the neural network model. This could be because the server lacks performance information for the operators corresponding to the neural network model, or the performance information for the operators in the server is incomplete. In this case, the terminal sends second performance information to the server, which can supplement the missing operator performance information. Of course, in other examples, the communication connection between the terminal and the server may fail, preventing the terminal from receiving feedback information. In this case, the terminal may choose not to send the second performance information to the server, or it may send the second performance information, but the server does not update its stored operator performance information based on the second performance information.

[0023] In one possible implementation of the first aspect, the terminal has an application and a target software development kit (SDK) installed, the application corresponding to the neural network model; the above steps include: sending a first request to the server, including: sending a second request to the target SDK through the application, the second request containing information about the neural network model; when the target SDK contains target configuration information, generating the first request based on the target configuration information and the second request, and the target SDK sending the first request to the server, the target configuration information indicating the information about the neural network model that the terminal can send to the server.

[0024] In this possible implementation, the target configuration information indicates that the function of optimizing model performance through edge-cloud interaction has been enabled. In this way, each application on the terminal can enable the function of optimizing model performance through edge-cloud interaction according to its own needs via the target SDK. This allows different applications on the terminal to interact with the server to obtain optimized neural network models, without needing to set up server interaction functions in each application during the development phase. This improves application development efficiency and facilitates the terminal's execution of operations related to server interaction.

[0025] In one possible implementation of the first aspect, the feedback information includes operator performance information and / or model operation strategy information for the neural network model. The operator performance information includes information on the running performance of the corresponding operator with respect to specified parameters, and the model operation strategy includes the first operation method of each of the multiple operators of the neural network model.

[0026] In this possible implementation, the specified parameters can be considered as factors affecting the performance of the operator. Different operators may have different specified parameters. For example, the specified parameters can be one or more of the following: operator parameters (e.g., the dimension of the input data, the dimension of the output data, the size of the convolution kernel), the version of the AI ​​inference framework, the type of computation unit, and the operating frequency of the computation unit.

[0027] In one possible implementation of the first aspect, the neural network model is not associated with the terminal hardware.

[0028] In this possible implementation, the neural network model not being associated with the terminal hardware means that the neural network model is not adapted for the terminal's chip or other hardware. This ensures the compatibility and universality of the neural network model when deployed on various devices. Furthermore, developers do not need to perform specific performance optimizations for different types and models of terminals during the development phase, reducing their workload and improving development efficiency.

[0029] A second aspect of this application provides a model performance optimization method applied to a server. The method includes: receiving a first request sent by a terminal, the first request carrying operator information of a neural network model and runtime environment information of the neural network model; sending feedback information to the terminal, the feedback information being obtained based on the operator information and runtime environment information, and the feedback information being able to determine a first operation mode for each of the multiple operators of the neural network model.

[0030] In the first aspect, after receiving the first request sent by the terminal, the server can query the operator performance information corresponding to the neural network model based on the operator information and runtime environment information carried in the first request. If the operator performance information corresponding to the neural network model is found, feedback information can be obtained based on the operator performance information corresponding to the neural network model.

[0031] In one possible implementation of the second aspect, the method further includes: receiving first performance information sent by a terminal, wherein the difference between the first performance information and the first reference information meets a preset condition, and the feedback information carries the first reference information; and updating the corresponding operator performance information in the server based on the first performance information.

[0032] In this possible implementation, when there is a significant difference between the operator performance information of the terminal during actual operation and the corresponding operator performance information in the server, the server can update the corresponding operator performance information stored in the server based on the operator performance information of the terminal during actual operation.

[0033] In one possible implementation of the second aspect, the method further includes: if the server does not receive feedback information, storing the operator information and runtime environment information of the neural network model.

[0034] In this possible implementation, when the server cannot obtain feedback information, it stores the operator information and runtime environment information of the neural network model, thereby indicating that the server lacks relevant operator performance information. Storing the operator information and runtime environment information of the neural network model allows the server to easily maintain and update the missing relevant operator performance information.

[0035] In one possible implementation of the second aspect, the method further includes: receiving second performance information sent by the terminal, the second performance information being used to indicate the operating performance of at least one operator with respect to the corresponding second operation mode; and updating the server based on the second performance information, the operator information of the neural network model, and the operating environment information.

[0036] In this possible implementation, the server can receive second performance information from the terminal to supplement the missing operator performance information in the server, thereby making the information stored in the server more complete.

[0037] A third aspect of this application provides a model performance optimization apparatus applied to a terminal. This apparatus has the function of implementing the method described in the first aspect or any possible implementation of the first aspect. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function, such as a transmitting module, a receiving module, and a processing module.

[0038] A fourth aspect of this application provides a terminal including at least one processor, a memory, and computer-executable instructions stored in the memory and executable on the processor. When the computer-executable instructions are executed by the processor, the processor executes a method as described in the first aspect or any possible implementation thereof.

[0039] The fifth aspect of this application provides a computer-readable storage medium storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor performs a method as described in the first aspect or any possible implementation thereof.

[0040] The sixth aspect of this application provides a computer program product that stores one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation thereof.

[0041] A seventh aspect of this application provides a chip system including a processor for supporting a terminal in implementing the functions described in the first aspect or any possible implementation thereof. In one possible design, the chip system may further include a memory for storing necessary program instructions and data for a computer device. This chip system may be composed of chips or may include chips and other discrete devices.

[0042] An eighth aspect of this application provides a model performance optimization apparatus, which can be applied to a server. This apparatus has the function of implementing the method described in the second aspect or any possible implementation of the second aspect. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function, such as a receiving module and a transmitting module.

[0043] The ninth aspect of this application provides a server including at least one processor, a memory, and computer-executable instructions stored in the memory and executable on the processor. When the computer-executable instructions are executed by the processor, the processor executes a method as described in the second aspect or any possible implementation thereof.

[0044] The tenth aspect of this application provides a computer-readable storage medium storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor performs a method as described in the second aspect above or any possible implementation thereof.

[0045] The eleventh aspect of this application provides a computer program product that stores one or more computer execution instructions, wherein when the computer execution instructions are executed by a processor, the processor executes a method as described in the second aspect above or any possible implementation thereof.

[0046] The twelfth aspect of this application provides a chip system including a processor for supporting a server in implementing the functions involved in the second aspect or any possible implementation thereof. In one possible design, the chip system may further include a memory for storing program instructions and data necessary for the computer device. This chip system may be composed of chips or may include chips and other discrete devices.

[0047] The technical effects of the third to seventh aspects or any of their possible implementations can be found in the first aspect or the related possible implementations of the first aspect, and the technical effects of the eighth to twelfth aspects or any of their possible implementations can be found in the second aspect or the related possible implementations of the second aspect, and will not be repeated here. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided in an embodiment of the present invention;

[0049] Figure 2 This is an exemplary schematic diagram of a server and terminal provided in an embodiment of the present invention;

[0050] Figure 3 This is an exemplary schematic diagram of the information interaction process between a server and a terminal provided in an embodiment of the present invention;

[0051] Figure 4 This is an exemplary schematic diagram of information interaction between a server and multiple terminals provided in an embodiment of the present invention;

[0052] Figure 5This is another exemplary schematic diagram of the information interaction process between the server and the terminal provided in the embodiments of the present invention;

[0053] Figure 6 This is another exemplary schematic diagram of the information interaction process between the server and the terminal provided in the embodiments of the present invention;

[0054] Figure 7 This is an exemplary schematic diagram of the information interaction process between the application, the target SDK, and the server provided in this embodiment of the invention;

[0055] Figure 8 This is a schematic diagram of an embodiment of the model performance optimization device provided in this application;

[0056] Figure 9 This is a schematic diagram of another embodiment of the model performance optimization device provided in this application;

[0057] Figure 10 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0058] Figure 11 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0059] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0060] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0061] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to those processes, methods, products, or apparatus.

[0062] Since the embodiments of this application involve the field of artificial intelligence, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0063] Please see Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0064] (1) Infrastructure

[0065] The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as central processing units (CPUs), graphics processing units (GPUs), neural network processing units (NPUs), tensor processing units (TPUs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs); the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0066] (2) Data

[0067] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0068] (3) Data processing

[0069] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0070] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training of data by symbolizing and formalizing it.

[0071] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0072] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0073] (4) General ability

[0074] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0075] (5) Smart Products and Industry Applications

[0076] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0077] (6) Neural Networks

[0078] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0079]

[0080] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0081] (7) Operator

[0082] An operator is a function that performs a specific task. For example, the reshape operator is used to reinterpret the shape of tensor data. Another example is the transpose operator, which is used to adjust the dimensional order of tensor data. In this application, commonly used functions for constructing deep learning model algorithms are collectively referred to as operators. Performing an operation on any function can be considered an operator. For example, convolution is a mathematical method of integral transformation. If a function f3 is generated by two functions f1 and f2, then f1, f2, and f3 can each be considered as an operator.

[0083] With the rapid development of AI, neural network models have been widely used in various devices such as mobile phones, televisions, desktop computers, and in-vehicle equipment to achieve functions such as speech recognition and image processing, greatly improving device performance and enhancing the user experience.

[0084] However, in actual deployment, the following problems often arise when assessing the performance of neural network models on the terminal:

[0085] (1) At present, the structures of neural network models used in various application scenarios are quite different. For example, in speech processing scenarios, the transformer model is often used, while in image processing scenarios, deep neural network (DNN), recurrent neural network (RNN) model or convolutional neural network (CNN) model is often used, which makes optimization more difficult.

[0086] (2) Different devices contain different computing units, and different computing units (e.g., CPU, GPU, NPU, TPU) are good at different computing scenarios. For example, NPU has better performance for matrix calculations, while CPU is better at logical operations and scalar calculations. If a neural network model with a large proportion of matrix calculations runs on an NPU, it can achieve several times to tens of times the performance improvement compared to running on a CPU. This results in a large performance difference when the same neural network model runs on different devices or computing units.

[0087] (3) The hardware and software versions in the device are often updated and iterated, and the running performance of the neural network model in the device is affected by the hardware and software versions. Therefore, in order to ensure the running performance of the neural network model in the device, the developers need to continuously optimize and adapt the neural network model as the hardware and software are updated and iterated. Otherwise, the running performance of the neural network model in the device may be greatly affected, thereby affecting the user experience.

[0088] Based on the above issues, in order to meet the deployment needs of applications on various devices and obtain better performance of neural network models on multiple devices during actual development, developers need to deploy neural network models on multiple devices and perform targeted performance optimizations on each device.

[0089] The following is a brief introduction to a commonly used model performance optimization method.

[0090] Currently, for a given device, after loading a neural network model, the device can parse the model and potentially perform simple optimizations, such as fusing or pruning operators. After these optimizations, operator information is obtained from the neural network model. This operator information is then distributed to the corresponding computing libraries of different computing units within the device, obtaining the computing unit support verification results returned by the libraries. These results may include performance information such as the computing unit's speed and power consumption in relation to the operators in the neural network model. Based on these results, the device can make decisions on how to allocate the operators in the neural network model to the device's computing units for computation. This allows the device to optimize model performance.

[0091] However, this common model performance optimization method for a specific device can lead to a long time consumption from loading the neural network model to applying the neural network model for computation. In addition, the model performance optimization strategy in the device is usually relatively simple, and the support verification results of the computing unit are usually few, resulting in less than ideal model performance optimization results.

[0092] Furthermore, the model parameters of a neural network model optimized for a specific device are deeply coupled with that device, resulting in poor compatibility between the optimized neural network model and other devices. Therefore, the optimization results for different devices are usually not universal, making the performance optimization of the neural network model for each device usually cumbersome and cannot be simplified.

[0093] It is evident that currently, optimizing the performance of neural network models in terminals is inefficient and requires a significant amount of work.

[0094] Based on this, this application provides a model performance optimization method to solve the problem that current developers need to deploy the trained neural network model on multiple devices and perform targeted testing and optimization on each device in order to obtain better performance of the neural network model on multiple devices, which results in low optimization efficiency.

[0095] The model performance optimization method of this application embodiment will be described in detail below.

[0096] The model performance optimization method of this application embodiment can be applied to an edge-cloud collaborative system, which includes a terminal and a server.

[0097] For example, the terminal can be a mobile phone, a tablet, a computer with wireless transceiver capabilities, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical care, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, or a wireless terminal in the Internet of Things (IoT), etc.

[0098] The server in this application embodiment can be a single server or a server cluster.

[0099] Data transmission between the terminal and the server can be conducted using a specified wireless or wired communication method. Various communication methods are possible between the terminal and the server, and this application does not limit the specific methods used in its embodiments.

[0100] like Figure 2 The diagram shown is an exemplary schematic of a server and a terminal.

[0101] Figure 2 In this context, server 201 can connect to multiple terminals (e.g., ...). Figure 2 The terminals 211, 212, and 213 shown interact with each other. The device types and models of terminals 211, 212, and 213 can be the same or different. The server 201 can also be connected to a data storage system.

[0102] The device has an application installed, which may contain a neural network model. This application may be downloaded and installed from an app store or other sources, or it may have come pre-installed with the device.

[0103] In some embodiments, the neural network model is not associated with the terminal hardware.

[0104] In other words, the neural network model is not adapted for the hardware such as the terminal's chip. This ensures the compatibility and universality of the neural network model when deployed on various devices. Furthermore, developers do not need to perform specific performance optimizations for different types and models of terminals during the development phase, reducing their workload and improving development efficiency.

[0105] This application does not limit the type of neural network model. For example, the neural network model may be a regression model, DNN, CNN, or RNN.

[0106] The neural network model for an application can be included in the application's installation package, or it can be downloaded to the terminal after the application is installed, based on the specified configuration information in the application. This neural network model can be used to process the data to be processed in the application.

[0107] This model performance optimization method can be applied to various information processing fields within the field of artificial intelligence, including natural language processing, image processing, and audio / video processing. The specific functions of this application and the neural network model are not limited here.

[0108] For example, if the application can perform facial recognition, it can include a neural network model for facial recognition. This model can process the image to be processed by the application to obtain the corresponding facial recognition result. In this case, the data to be processed in the application is the image to be processed. Alternatively, if the application can perform text translation, the neural network model can translate the text information obtained by the application to obtain the translation result. In this case, the data to be processed in the application is the text information.

[0109] refer to Figure 3 Specifically, the model performance optimization method in this application embodiment may include steps 301-305.

[0110] Step 301: The terminal sends the first request to the server.

[0111] The first request carries operator information of the neural network model and runtime environment information of the neural network model.

[0112] In this embodiment of the application, the server can provide a specified application programming interface (API), and the terminal calls the API and sends a first request to the server based on the communication connection with the server.

[0113] Generally speaking, operator information can include information about each operator in a neural network model.

[0114] The number and type of operators in a neural network model can be determined based on the actual application scenario and are not limited here. For example, operators in a neural network model can include one or more of the following: convolution operators, pooling operators, activation operators, and fully connected operators. Furthermore, each type of operator can include one or more operators, and the operator parameters of operators of the same type can be the same or different. For instance, a neural network model can include two convolution operators, two pooling operators, and one fully connected operator. Operators in a neural network model can correspond to layers in a neural network.

[0115] Operator information may include one or more of the following information for the corresponding operator: operator type, operator weight information, and one or more operator parameter information. This operator parameter information differs from the operator weights; for example, it may include the dimensions of the operator's input and / or output data, and information about the relevant parameters of the elements in the operator (e.g., the size of the convolution kernel in a convolution operator).

[0116] In addition, in some examples, the first request may also include information on the relationships between operators of the neural network model.

[0117] The relationships between operators can include the connections between various operators in the neural network model and the data flow between them. These relationships are represented by the graph structure of the neural network model.

[0118] The specific form of the operator information and the relationship information between operators in the first request is not limited here.

[0119] For example, the first request may record the operator information of each operator in the neural network model in the form of a list. Alternatively, the first request may contain a complete neural network model, in which case the complete neural network model contains not only the operator information of each operator, but also the relationship information between operators.

[0120] The runtime environment information corresponding to the neural network model can include the hardware and / or software information associated with the neural network model when it is running on the terminal.

[0121] For example, the software information may include information about the application corresponding to the neural network model and / or information about the AI ​​inference framework. The AI ​​inference framework controls the operation of the neural network model on the terminal. For instance, the AI ​​inference framework can load and parse the neural network model, obtain information about the operators in the neural network model, then obtain the binary executable files corresponding to the operators of the neural network model, and then allocate them to the terminal's computing unit for computation. The application information may include one or more of the following: the application's name, identifier, version number, etc. The AI ​​inference framework information may include one or more of the following: the AI ​​inference framework's name, identifier, version number, etc.

[0122] The hardware information may include information about the chip corresponding to the neural network model in the terminal. This chip information may include one or more of the following: chip supplier, chip model, and information about the computing units contained in the chip.

[0123] The computing units in a chip can include CPUs, or one or more of GPUs, NPUs, and TPUs. Information about the computing units contained in the chip can include one or more of the following: the type and model of the computing unit, performance parameters (such as the operating frequency of the computing unit), and the support for operators by the computing unit.

[0124] In addition, the first request may also carry identification information such as the terminal's name and International Mobile Equipment Identity (IMEI) so that the server can identify the sender of the first request.

[0125] In this embodiment of the application, there may be a variety of triggering conditions for the terminal to send the first request to the server, which are not limited here.

[0126] For example, the neural network model corresponds to an application. This could be achieved by the terminal sending a first request to the server each time the application is launched; or by sending the first request to the server when the application is launched for the first time; or by sending the first request to the server when the application is launched and the neural network model is first applied; or by sending the first request to the server when the application is launched and it is detected that the terminal does not have information on the optimized neural network model stored in the terminal; or by periodically triggering the operation of sending the first request to the server while the application is running; or by sending the first request to the server after detecting updated information about the neural network model.

[0127] Step 302: The server receives the first request sent by the terminal.

[0128] After receiving the first request from the terminal, the server can query the operator performance information corresponding to the neural network model based on the operator information and runtime environment information carried in the first request. If the operator performance information corresponding to the neural network model is found, feedback information can be obtained based on the operator performance information corresponding to the neural network model.

[0129] For example, the server may include operator performance information and index information corresponding to each operator performance information. In this way, the operator information and runtime environment information carried in the first request can be matched with each index information. If the operator information and runtime environment information carried in the first request successfully match a certain index information, the operator performance information corresponding to that successfully matched index information can be used as the operator performance information corresponding to the neural network model, thereby obtaining feedback information.

[0130] The operator performance information includes information on the performance of the corresponding operator with respect to specified parameters.

[0131] Specified parameters can be considered factors affecting the performance of operators. Since operator-related operations are performed by computing units in the terminal, specified parameters typically include parameters related to the computing units. Different operators may have different specified parameters. For example, specified parameters can be one or more of the following: the data dimension of the operator (e.g., the dimension of the input data, the dimension of the output data), operator parameters, the version of the AI ​​inference framework, the type of computing unit, and the operating frequency of the computing unit.

[0132] Operator parameters are related to the type of operator; different types of operators can have different operator parameters. Taking convolution operators as an example, the operator parameters can include the size of the convolution kernel and the stride of the kernel during the convolution operation. Generally speaking, although current neural network models have a wide variety of types and structures, the types of operators involved are relatively limited. The range of operator parameters and data dimensions for the same type of operator is also limited, and in many cases, they may be the same. Therefore, operator performance information for any type of operator can usually be used in multiple neural network models that include that operator.

[0133] Step 303: If the server obtains feedback information for the first request based on the operator information and the runtime environment information, then the server sends the feedback information to the terminal.

[0134] Feedback information can be used to determine the first operation for each of the multiple operators in a neural network model.

[0135] The first operation mode is used to indicate how the operating parameters of the operator are set when performing operations at the terminal. These operating parameters may include parameters that affect the operational performance of the corresponding operator. For example, these operation parameters may include parameters related to the computing unit.

[0136] In some embodiments, the first operation mode of each of the plurality of operators includes the operation mode of the corresponding operator in the corresponding first computing unit, wherein the first computing unit is one or more computing units of the terminal.

[0137] Since the operations corresponding to operators in a terminal are usually executed by the computing unit in the terminal, it can be considered that the first operation mode of each operator in the multiple operators of the neural network model specifically refers to:

[0138] The first operation mode of the first computational unit corresponding to each of the multiple operators in the neural network model.

[0139] At this time, the first operation mode is used to indicate how the operating parameters of the operator are set when it performs operations in the corresponding first computing unit. For example, the operating parameters may include the operating frequency of the computing unit.

[0140] The first computing unit can be one or more computing units in the terminal. When there are multiple first computing units in the terminal, the types of each first computing unit can be the same or different, and the hardware configurations of each first computing unit can be the same or different.

[0141] The feedback information can be used to determine the first operation mode of each operator among the multiple operators of the neural network model. Specifically, the feedback information can be used to determine the first computation unit corresponding to each operator among the multiple operators of the neural network model, and the first operation mode of each operator among the multiple operators of the neural network model in the corresponding first computation unit.

[0142] At this point, the feedback information can include either of the following two scenarios:

[0143] (1) The feedback information may include information on the model operation strategy corresponding to the neural network model. The model operation strategy includes the first computation unit corresponding to each of the multiple operators of the neural network model, and the first operation mode of each of the multiple operators of the neural network model in the corresponding first computation unit.

[0144] (2) The feedback information includes the operator performance information corresponding to the neural network model, and the operator performance information included in the feedback information satisfies the specified optimization conditions.

[0145] In this scenario, the fact that the operator performance information included in the feedback information meets the specified optimization conditions can be understood as the feedback information containing sufficient operator performance information. In one example, determining that the operator performance information included in the feedback information meets the specified optimization conditions could mean that the operator performance information contained in the feedback information is related to each computing unit of the terminal. This enables the terminal to determine, from its computing units, the computing unit with superior performance for the corresponding operator as the first computing unit for that operator, and to determine the operation method with superior performance for the corresponding operator as the first operation method for that operator.

[0146] In some embodiments, the feedback information includes operator performance information and / or model operation strategy information for the neural network model. The operator performance information includes information on the running performance of the corresponding operator with respect to specified parameters, and the model operation strategy includes the first operation method of each of the multiple operators of the neural network model.

[0147] In one example, the feedback information may include operator performance information of the neural network model. The operator performance information corresponding to the neural network model can indicate the performance of the operators in the neural network model when they are operated in at least one operational manner.

[0148] In another example, the feedback information may include information about the first operation method corresponding to each of the multiple operators in the neural network model.

[0149] Step 304: The terminal receives feedback information sent by the server within the first time.

[0150] The time interval between the first time and the time of sending the first request can be determined according to the actual application scenario. For example, the time interval between the first time and the time of sending the first request can be pre-configured by the developer on the terminal; or, the server can pre-determine and send it to the terminal based on information such as the time required for the server to obtain feedback information, and then the terminal can configure it.

[0151] If the terminal does not receive feedback from the server's first request within the first time, or if the feedback received by the terminal is insufficient to determine the first operation method of each of the multiple operators in the neural network model, then refer to... Figure 5 The description of step 503 and subsequent steps in the relevant embodiments will not be repeated here.

[0152] Feedback information can be used to determine the first operation of each of the multiple operators in a neural network model, and can include either of the following two cases:

[0153] (1) The feedback information may include information about the model operation strategy corresponding to the neural network model.

[0154] The model's computational strategy includes the first computational method corresponding to each of the multiple operators in the neural network model.

[0155] At this point, based on the feedback information, the terminal can directly obtain information about the first operation method corresponding to each of the multiple operators in the neural network model.

[0156] (2) The feedback information includes the operator performance information corresponding to the neural network model, and the operator performance information included in the feedback information satisfies the specified optimization conditions.

[0157] In this case, the fact that the operator performance information included in the feedback information satisfies the specified optimization conditions can be understood as the feedback information including sufficient operator performance information.

[0158] For the terminal, there are multiple ways to determine whether the operator performance information included in the feedback information meets the specified optimization conditions.

[0159] In one example, the feedback information may include operator performance information for each operator of the neural network model, and the operator performance information contained in the feedback information is related to each computing unit of the terminal, thereby enabling the terminal to determine from the computing units of the terminal that have better performance for the corresponding operator as the first computing unit of the corresponding operator, and to determine the operation mode that has better performance for the corresponding operator as the first operation mode of the corresponding operator.

[0160] For example, neural network models contain convolution operators, and terminal chips contain CPUs, NPUs, and GPUs.

[0161] If the feedback information includes the convolution operator's performance information on the CPU and the NPU, but does not include the convolution operator's performance information on the GPU, then it can be considered that the feedback information regarding the convolution operator's performance is incomplete and does not include sufficient operator performance information. Therefore, it can be considered that the operator performance information in the feedback information does not meet the specified optimization conditions.

[0162] If the feedback information includes the convolution operator's performance information on the CPU, the NPU, and the GPU, then it can be considered that the feedback information contains relatively complete information on the convolution operator's performance, meets the specified optimization conditions, and therefore contains sufficient information on the convolution operator's performance.

[0163] At this point, the terminal can compare the convolution operator's performance information on the CPU, the NPU, and the GPU, and select the computing unit with the best performance as the first computing unit for the convolution operator.

[0164] In one example, the determination can be for each of the multiple operators in the neural network model, determining the first computational unit corresponding to each operator, and the first operation method of each operator in the corresponding first computational unit.

[0165] In another example, the operators of the neural network model can be combined to obtain N sets of first operators. Then, the first computational unit corresponding to each set of first operators and the first operation method of each set of first operators in its corresponding first computational unit are determined, where N is a positive integer and each set of first operators contains at least one operator. There are various strategies for combining the operators, which are not limited here. For example, the operators can be combined based on their type, number, and relationships to obtain N sets of first operators. For instance, if the neural network model includes multiple adjacent convolution operators, and the operator parameters such as operator dimension of these adjacent convolution operators are the same or similar, then these adjacent convolution operators can be combined to obtain a set of first operators. The convolution operators in this set of first operators can be run by the same first computational unit.

[0166] Generally speaking, the performance of a neural network model on a terminal is largely determined by the performance of each of its multiple operators. Since operators in a neural network model are executed by different computational units, the impact of execution across different computational units is usually relatively small and not a decisive factor in the model's performance. In other words, if each operator in the neural network model is assigned to the computational unit with the best performance, then the overall performance of the neural network model on the terminal is usually optimal or relatively good.

[0167] Therefore, in this embodiment of the application, for the neural network model of the terminal, the optimal operation mode (i.e., the first operation mode) of each operator among the multiple operators of the neural network model can be obtained based on the relevant feedback information of the server. This enables the terminal to achieve better running performance when performing neural network operations in the first operation mode, without requiring developers to perform complex performance optimization operations for various types and models of devices that the neural network model may be deployed on, thereby improving the optimization efficiency of the neural network model on the terminal.

[0168] In some embodiments, the method further includes step 305.

[0169] Step 305: Based on the feedback information, the terminal obtains the first calculation result of the data to be processed.

[0170] The data to be processed can come from the application corresponding to the neural network model. The type of data to be processed is not limited here. For example, the application can implement facial recognition; in this case, the data to be processed in the application is an image. Alternatively, the application can implement text translation; in this case, the data to be processed in the application is text information.

[0171] In this embodiment, based on the first computing unit and first operation method corresponding to each operator in the neural network model, the corresponding operators can be compiled to obtain binary files that can be efficiently loaded and run by the corresponding first computing unit. Thus, when data to be processed is obtained, the runtime sequence of each first computing unit can be determined according to the structure of the neural network model, and based on this runtime sequence, the binary files of the corresponding operators can be loaded and run by each first computing unit to obtain the first running result for the data to be processed.

[0172] In some examples, after obtaining the binary file for each operator that can be efficiently loaded and run by the corresponding first computing unit, the terminal can store the binary file. This facilitates efficient loading and running by the first computing unit when processing instructions for each set of data are received multiple times in the future. Furthermore, after being stored on the terminal, the binary file can be repeatedly loaded and run as needed during the current run of the application corresponding to the neural network model, and also in the next and subsequent runs of the application.

[0173] In other examples, after obtaining the binary file for each operator that can be efficiently loaded and run by the corresponding first computing unit, due to the large size of the binary file or the limited storage resources in the terminal, the terminal only temporarily stores the binary file in memory or other means during the current run of the application corresponding to the neural network model. After the application finishes running, the binary file is deleted from the terminal. In this example, the terminal can trigger the operation of sending a first request to the server and subsequent operations each time the application is launched, so as to redetermine the first computing unit and the corresponding first running mode for each operator in the neural network model each time the application is launched, and obtain the corresponding binary file for the first computing unit to load and run.

[0174] In this embodiment of the application, the server can interact with multiple terminals to optimize the performance of the neural network models of the multiple terminals respectively.

[0175] For example, such as Figure 4 The diagram shown is an exemplary illustration of information interaction between a server and multiple terminals.

[0176] Server 40 can interact with terminals 41, 42, and 43 respectively to provide model performance optimization functions for terminals 41, 42, and 43. Specifically, the operators in the neural network model 10 of application 1 in terminal 41 can be computed using CPU 411, GPU 412, and NPU 413 respectively; the operators in the neural network model 20 of application 2 in terminal 42 can be computed using CPU 421 and GPU 422 respectively; and the operators in the neural network model 30 of application 3 in terminal 43 can be computed using CPU 431 and NPU 432 respectively.

[0177] As can be seen, through the embodiments of this application, the server can efficiently and quickly provide a better running mode for the neural network models corresponding to multiple terminals, resulting in high optimization efficiency.

[0178] In some embodiments, such as Figure 5 As shown, after step 302 above, the model performance optimization method further includes:

[0179] Step 503: If the terminal does not receive feedback information within the first time, or if the feedback information cannot be used to determine the first operation method of each operator among the multiple operators of the neural network model, then the second operation result of the data to be processed is obtained based on the second operation method of each operator among the multiple operators of the preset neural network model.

[0180] Specifically, the second operation method of each operator in the multiple operators of the neural network model can refer to the second operation method of each operator in the corresponding second computation unit.

[0181] The second computing unit can be one or more computing units in the terminal. When there are multiple second computing units in the terminal, the types of the various second computing units can be the same or different, and the hardware configurations of the various second computing units can be the same or different.

[0182] It is understood that the second computing unit and the first computing unit in the embodiments of this application both belong to the computing units of the terminal. The determination strategy of the first computing unit and the determination strategy of the second computing unit are usually independent of each other. Therefore, the first computing unit and the second computing unit can be the same or different. The first operation method corresponding to the first computing unit and the second operation method corresponding to the second computing unit can be the same or different, which is not limited here.

[0183] There could be several reasons why the terminal did not receive the feedback information. For example, there might be a communication failure between the terminal and the server, or the server might not have sent the feedback information.

[0184] The fact that the feedback information received by the terminal in the first instance cannot be used to determine the first operation method corresponding to each of the multiple operators in the neural network model may be due to the slow transmission speed of the communication connection between the terminal and the server, which causes the terminal to fail to fully receive the feedback information sent by the server, or it may be that the feedback information sent by the server cannot be used to determine the first operation method corresponding to each of the multiple operators in the neural network model.

[0185] The feedback information sent by the server cannot be used to determine the first operation mode of each operator among multiple operators in the neural network model. This feedback information may not support the terminal in determining the first operation mode corresponding to all operators in the neural network model.

[0186] For a specific operator in a neural network model, the feedback information may not be sufficient to determine the first operation method corresponding to that operator. This could be because the feedback information does not contain information about that operator, or because the operator performance information contained in the feedback information is incomplete. For example, it may lack performance information about certain computing units and / or certain operation methods in the terminal. In this case, it's possible that the server has not found complete operator performance information corresponding to multiple operators in the neural network model, or that the terminal has not received the complete information sent by the server in the first instance.

[0187] In this embodiment, the information of the second operation method corresponding to each of the multiple operators in the neural network model can be pre-stored in the terminal, or it can be determined by the terminal based on a preset operation strategy, according to the operator information and the operating environment information of the neural network model. The preset operation strategy is not limited here.

[0188] Through the embodiments of this application, when the terminal fails to obtain the running strategy for the neural network model from the server, it can determine the second operation mode corresponding to each of the multiple operators of the neural network model based on the model running strategy already in the terminal, thereby optimizing the running performance of the neural network model.

[0189] The next time the neural network model is loaded, the operation of sending the first request to the server and subsequent operations can be re-executed to try to optimize the performance of the neural network model again.

[0190] In some embodiments, the server stores information on the correspondence between index information and operator performance information. The index information is used to match with operator information and runtime environment information, and the operator performance information is used to obtain feedback information when the corresponding index information matches the operator information and runtime environment information successfully.

[0191] In one example, the correspondence between the index information and the operator performance information can be stored in the form of a relation list. Other storage formats for this correspondence are also possible and are not limited here.

[0192] The following section uses a relation list as an example to introduce the information stored in the server.

[0193] In this example, the relationship list indicates the correspondence between index information and operator performance information. Typically, each index in the relationship list corresponds one-to-one with each operator performance information. An index and its corresponding operator performance information can form a set of correspondences. The relationship list can include one or more sets of correspondences.

[0194] For example, the index information may include reference operator information and reference runtime environment information for the corresponding operator, which can be used to match the operator information and runtime environment information in the first request.

[0195] Reference operator information includes one or more of the following:

[0196] Operator type, operator weight information, one or more operator parameter information (e.g., the dimension of the input data of the operator, the dimension of the output data, the size of the convolution kernel in the convolution operator, etc.), and the relationship information between operators.

[0197] The reference operating environment information includes reference hardware operating environment information and reference software operating environment information.

[0198] The reference hardware operating environment information includes one or more of the following:

[0199] The chip supplier, chip model, type of computing unit contained in the chip, model of computing unit, performance parameters (e.g., operating frequency of computing unit), and support for operators by computing unit.

[0200] The reference software runtime environment information includes one or more of the following:

[0201] Information about the application corresponding to the neural network model (e.g., application name, identifier, and / or version number) and information about the AI ​​inference framework (e.g., AI inference framework name and version number).

[0202] When a certain index information successfully matches the operator information and runtime environment information in the first request, the operator performance information corresponding to the successfully matched index information can be used to obtain feedback information. Specifically, a successful match between a certain index information and the operator information and runtime environment information in the first request means that the operator information and runtime environment information in the first request find matching information in that index information.

[0203] When all index information in the relation list fails to match the operator information and runtime environment information in the first request, it can be assumed that the server cannot obtain feedback information.

[0204] In this embodiment of the application, through the relationship list, the server can quickly and efficiently query the operator performance information of the operators in the neural network model associated with the first request, and obtain feedback information when the operator performance information of the operators in the neural network model is queried.

[0205] Furthermore, the server can maintain and update the operator performance information and the corresponding index information stored in the server.

[0206] The following sections will describe in detail how to maintain and update this relationship list for several scenarios.

[0207] 1. In one scenario, when the server cannot find the operator performance information corresponding to the neural network model of the first request in the server, it updates the corresponding information in the server based on the second performance information subsequently uploaded by the terminal.

[0208] Specifically, such as Figure 5 As shown, in some embodiments, the method further includes steps 504-507:

[0209] Step 504: If the server does not receive feedback information, it stores the operator information and operating environment information of the neural network model.

[0210] The server stores operator information and runtime environment information of the neural network model, thereby indicating that the server lacks relevant operator performance information.

[0211] The server stores operator information and runtime environment information of the neural network model, which facilitates the maintenance and updating of missing related operator performance information. For example, it can supplement the information with secondary performance information and related index information sent by the terminal later, or it can supplement this part of operator performance information and corresponding index information manually.

[0212] Step 505: After obtaining the second calculation result, the terminal sends the second performance information to the server.

[0213] The second performance information is used to indicate the performance of at least one operator with respect to the corresponding second operation mode.

[0214] Step 506: The server receives the second performance information sent by the terminal.

[0215] Step 507: The server updates the server based on the second performance information, the operator information of the neural network model, and the operating environment information.

[0216] In this embodiment, if the terminal fails to obtain feedback information from the server regarding the first operation method corresponding to each of the multiple operators in the neural network model, this may be because the server lacks operator performance information corresponding to the neural network model, or the operator performance information for the neural network model in the server is incomplete. Therefore, after obtaining the second operation result, the terminal can send second performance information to the server. This second performance information indicates the performance of all operators in the neural network model with respect to the corresponding second operation method, or it can indicate only the performance of some operators in the neural network model with respect to the corresponding second operation method. The second performance information can be operator performance information not involved in the feedback information, i.e., operator performance information that the server may lack.

[0217] In this way, the server can receive second performance information from the terminal to supplement the missing operator performance information in the server, thereby making the corresponding information in the server more complete. Specifically, index information can be obtained based on the operator information and runtime environment information of the neural network model, and the corresponding operator performance information can be obtained based on the second performance information. Then, the index information and the corresponding operator performance information are added as a correspondence to the relationship list.

[0218] Of course, in some cases, the communication connection between the terminal and the server may fail, causing the terminal to be unable to receive feedback information. In this case, the terminal may choose not to send the second performance information to the server, or the terminal may send the second performance information to the server, but the server may not update the stored operator performance information based on the second performance information.

[0219] 2. In another scenario, when there is a significant difference between the operator performance information of the terminal during actual operation and the corresponding operator performance information in the relation list, the server can update the corresponding operator performance information in the server based on the operator performance information of the terminal during actual operation.

[0220] Specifically, in some embodiments, the feedback information also carries first reference information;

[0221] like Figure 6 Following step 305 above, the model performance optimization method further includes steps 601-603:

[0222] Step 601: After obtaining the first calculation result, if the difference between the first performance information and the first reference information meets the preset conditions, the terminal sends the first performance information to the server.

[0223] The first performance information is used to indicate the performance of at least one operator with respect to the corresponding first operation mode.

[0224] Step 602: The server receives the first performance information sent by the terminal.

[0225] Step 603: Based on the first performance information, the server updates the corresponding operator performance information in the server.

[0226] In this embodiment of the application, the first reference information may indicate the expected operating performance corresponding to the corresponding operator. This expected operating performance is the anticipated operating performance when the corresponding operator's operation is performed using the first operating method.

[0227] In this embodiment of the application, the difference between the first performance information and the first reference information meeting a preset condition can indicate that the difference between the first performance information and the first reference information is large.

[0228] The preset conditions can be determined based on how the performance is described.

[0229] For example, in the first performance information, the running performance of the operator is described by the corresponding runtime. In the first reference information, the expected running performance of the operator is also described by the corresponding runtime. The difference between the first performance information and the first reference information can be represented by the difference or error rate between the runtime in the first performance information and the runtime in the first reference information. If the difference or error rate is greater than the preset deviation threshold, it can be considered that the difference between the first performance information and the first reference information meets the preset conditions.

[0230] In real-world scenarios, when there is a significant difference between the operator performance information corresponding to the terminal during actual operation and the operator performance information corresponding to the relation list, it may be due to a large error in the relevant operator performance information stored in the server. Therefore, sending the first performance information from the terminal to the server can provide data reference for the server and promptly correct the operator performance information with large errors in the server.

[0231] For example, the first reference information includes a reference runtime of 3 milliseconds for the convolution operator to be operated on in the GPU at a first frequency. However, in the terminal, when obtaining the first calculation result, the actual runtime of the terminal's GPU operating on the convolution operator at the first frequency is 2 milliseconds. In this case, the error rate of the actual runtime relative to the reference runtime is 50%, which is greater than the preset deviation threshold of 20%. Therefore, it can be considered that the difference between the terminal's GPU's performance operating on the convolution operator at the first frequency and the first reference information meets the preset conditions. Thus, the terminal can use the GPU's performance operating on the convolution operator as the first performance information and send it to the server, which then changes the corresponding operator performance information stored in the server to 2 milliseconds.

[0232] In some examples, after receiving the first performance information, the server may not immediately update the corresponding information in the server based on the first performance information. Instead, the server may update the corresponding operator performance information in the server only when it receives the actual performance information for the same operator from multiple terminals. In this way, the relevant operator performance information can be authenticated multiple times by multiple terminals, and the server will only update the corresponding operator performance information in the server when it is determined that the relevant operator performance information in the server has a large error.

[0233] 3. Operator performance information and corresponding index information in the server can be updated manually.

[0234] When the chip and / or software versions in the terminal are updated, the neural network models in the application are updated, or there are new optimizations to the operator's operation methods, the operator performance data in the server may no longer meet the terminal's needs. In this case, maintenance personnel can update the operator performance data and corresponding index information stored in the server. Specifically, based on the situation to be updated, test cases can be constructed and operator performance tests can be performed to collect new operator performance information, which can then be used to update the relevant information in the server.

[0235] Based on any embodiment of the above model performance optimization method, the internal processing flow of the terminal involved in the model performance optimization method will be described in detail below.

[0236] In some embodiments, the terminal has an application and a target software development kit (SDK) installed. The application, corresponding to the neural network model, sends a first request to the server, including:

[0237] The application sends a second request to the target SDK, the second request containing information about the neural network model;

[0238] When the target SDK contains target configuration information, a first request is generated based on the target configuration information and the second request, and the target SDK sends the first request to the server. The target configuration information is used to indicate the information content about the neural network model that the terminal can send to the server.

[0239] like Figure 7 The diagram shown is an exemplary schematic of the information interaction process between the application in the terminal, the target software development kit (SDK), and the server.

[0240] In this embodiment of the application, the target configuration information is used to indicate that the function of optimizing model performance through end-to-cloud interaction has been enabled.

[0241] In this case, the application can enable the model performance optimization function based on end-to-cloud interaction in any of the above embodiments through the target SDK. At this time, the target SDK can obtain the target configuration information.

[0242] After enabling the model performance optimization function based on edge-cloud interaction, the application can send a second request to the target SDK through the API provided by the target SDK. This second request includes information about the neural network model.

[0243] The target SDK generates a first request based on the target configuration information and the neural network model information, and can send this first request to the server through the target SDK. Then, the target SDK can receive feedback information sent by the server, thereby enabling the terminal to perform subsequent operations based on the feedback information.

[0244] In this embodiment of the application, the target SDK can be used to provide different applications in the terminal with the function of interacting with the server to obtain the optimized neural network model, without the need to set up the relevant functions of interacting with the server in different applications during the development stage, thereby improving the development efficiency of the application and facilitating the terminal to perform the relevant operations of interacting with the server.

[0245] The above embodiments of this application have described the model performance optimization method from multiple aspects. The following, in conjunction with the accompanying drawings, describes the model performance optimization device applied to a terminal and the model performance optimization device applied to a server.

[0246] like Figure 8 As shown, this application embodiment provides a model performance optimization device 80, which can be applied to the terminal in the above embodiment.

[0247] One embodiment of the device 80 includes:

[0248] The sending module 801 is used to send a first request to the server. The first request carries operator information of the neural network model and runtime environment information of the neural network model.

[0249] The receiving module 802 is used to receive feedback information sent by the server in response to the first request within a first time. The feedback information can be used to determine the first operation mode of each of the multiple operators in the neural network model.

[0250] Optionally, the feedback information also includes first reference information;

[0251] The sending module 801 is also used for:

[0252] After obtaining the first calculation result, if the difference between the first performance information and the first reference information meets the preset conditions, the first performance information is sent to the server, wherein the first performance information is used to indicate the running performance of at least one operator with respect to the corresponding first calculation method.

[0253] Optionally, the device 80 also includes a processing module 803;

[0254] Processing module 803 is used for:

[0255] If no feedback information is received in the first time, or if the feedback information cannot be used to determine the first operation method of each of the multiple operators in the neural network model, then the second operation result of the data to be processed is obtained based on the second operation method of each of the multiple operators in the preset neural network model.

[0256] Optionally, the transmitting module 801 is also used for:

[0257] After obtaining the second operation result, second performance information is sent to the server. The second performance information is used to indicate the running performance of at least one operator with respect to the corresponding second operation method.

[0258] Optionally, the terminal is equipped with an application and a target software development kit (SDK), with the application corresponding to the neural network model;

[0259] The sending module 801 is used for:

[0260] The application sends a second request to the target SDK, the second request containing information about the neural network model;

[0261] When the target SDK contains target configuration information, a first request is generated based on the target configuration information and the second request, and the target SDK sends the first request to the server. The target configuration information is used to indicate the information content about the neural network model that the terminal can send to the server.

[0262] like Figure 9 As shown, this application embodiment provides a model performance optimization device 90, which can be applied to the server in the above embodiment.

[0263] One embodiment of the device 90 includes:

[0264] The receiving module 901 is used to receive a first request sent by the terminal. The first request carries operator information of the neural network model and operating environment information of the neural network model.

[0265] The sending module 902 is used to send feedback information to the terminal. The feedback information is obtained based on operator information and operating environment information. Furthermore, the feedback information can be used to determine the first operation mode of each of the multiple operators in the neural network model.

[0266] Optionally, the device 90 also includes an update module 903;

[0267] The receiving module 901 is also used to receive first performance information sent by the terminal, wherein the difference between the first performance information and the first reference information meets a preset condition, and the feedback information carries the first reference information;

[0268] The update module 903 is used to update the corresponding operator performance information in the server based on the first performance information.

[0269] Optionally, the device 90 also includes a storage module 904;

[0270] Storage module 904 is used to store operator information and runtime environment information of the neural network model if the server does not receive feedback information.

[0271] Optionally, the receiving module 901 is further configured to receive second performance information sent by the terminal, the second performance information being used to indicate the operating performance of at least one operator with respect to the corresponding second operation mode;

[0272] The update module 903 is used to update the server based on the second performance information, the operator information of the neural network model, and the runtime environment information.

[0273] Figure 10 The diagram shown is a possible logical structure of a terminal 100 provided in an embodiment of this application. The terminal 100 is used to implement the functions of the terminal involved in any of the above embodiments. The terminal 100 includes: a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004. The memory 1001, processor 1002, and communication interface 1003 are interconnected via the bus 1004.

[0274] The memory 1001 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1001 can store programs. When the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to execute steps 301, 304-305, 503, 505, 601, etc., of the above-described model performance optimization method embodiment.

[0275] The processor 1002 can be a central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), digital signal processor (DSP), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any combination thereof. It is used to execute relevant programs to achieve the functions required by the sending module, receiving module, and processing module in the model performance optimization device applied to the terminal in the above embodiments, or to execute steps 301, 304-305, 503, 505, 601, etc., of the model performance optimization method embodiments of this application. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1001. Processor 1002 reads the information in memory 1001 and executes steps 301, 304-305, 503, 505, 601, etc. of the above-mentioned model performance optimization method embodiment in conjunction with its hardware.

[0276] The communication interface 1003 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the terminal 100 and other devices or communication networks. For example, the communication interface 1003 can be used to exchange information with a server.

[0277] Bus 1004 enables the transmission of information between various components of terminal 100 (e.g., memory 1001, processor 1002, and communication interface 1003). Bus 1004 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0278] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device performs the aforementioned... Figure 10 The steps performed by the processor in the process.

[0279] In another embodiment of this application, a computer program product is also provided, which includes computer-executable instructions stored in a computer-readable storage medium; when the processor of the device executes the computer-executable instructions, the device performs the above-described... Figure 10 The steps performed by the processor in the process.

[0280] In another embodiment of this application, a chip system is also provided, the chip system including a processor for implementing the above. Figure 10 The steps performed by the processor. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for data writing. The chip system may consist of chips or may include chips and other discrete components.

[0281] Figure 11 The diagram shown is a possible logical structure of a server 110 provided in an embodiment of this application. The server 110 is used to implement the functions of the server involved in any of the above embodiments. The server 110 includes: a memory 1101, a processor 1102, a communication interface 1103, and a bus 1104. The memory 1101, processor 1102, and communication interface 1103 are interconnected via the bus 1104.

[0282] The memory 1101 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1101 can store programs. When the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 are used to execute steps 302-303, 504, 506-507, 602-603, etc., of the above-described model performance optimization method embodiment.

[0283] The processor 1102 can be a central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), digital signal processor (DSP), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any combination thereof, to execute relevant programs to achieve the functions required by the receiving module, transmitting module, storage module, and updating module in the model performance optimization device applied to the server in the above embodiments, or to execute steps 302-303, 504, 506-507, 602-603, etc., of the model performance optimization method embodiments of this application. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of the hardware decoding processor, or the execution of the hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 1101. Processor 1102 reads the information in memory 1101 and executes steps 302-303, 504, 506-507, 602-603, etc. of the above-mentioned model performance optimization method embodiment in conjunction with its hardware.

[0284] The communication interface 1103 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the server 110 and other devices or communication networks. For example, it can interact with the terminal involved in any of the above embodiments.

[0285] Bus 1104 enables the transmission of information between various components of server 110 (e.g., memory 1101, processor 1102, and communication interface 1103). Bus 1104 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0286] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device performs the aforementioned... Figure 11 The steps performed by the processor in the process.

[0287] In another embodiment of this application, a computer program product is also provided, which includes computer-executable instructions stored in a computer-readable storage medium; when the processor of the device executes the computer-executable instructions, the device performs the above-described... Figure 11 The steps performed by the processor in the process.

[0288] In another embodiment of this application, a chip system is also provided, the chip system including a processor for implementing the above. Figure 11 The steps performed by the processor. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for data writing. The chip system may consist of chips or may include chips and other discrete components.

[0289] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0290] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0291] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0292] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0293] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0294] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this application, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0295] The above are merely specific implementation methods of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto.

Claims

1. A model performance optimization method, characterized by, The method applied to a terminal comprises: sending a first request to a server, the first request carrying operator information of a neural network model and running environment information of the neural network model; receiving feedback information sent by the server for the first request within a first time, the feedback information being used to determine a first operation mode of each operator in the neural network model; the terminal is installed with an application program and a target software development kit (SDK), the application program corresponding to the neural network model; the sending of the first request to the server comprises: sending a second request to the target SDK through the application program, the second request containing information of the neural network model; when the target SDK contains target configuration information, generating the first request based on the target configuration information and the second request, and sending the first request to the server by the target SDK, the target configuration information being used to indicate information content about the neural network model that the terminal can send to the server.

2. The method of claim 1, wherein, The method further comprises: obtaining a first operation result of to-be-processed data based on the feedback information.

3. The method of claim 1, wherein, The first operation mode of each operator in the neural network model comprises an operation mode of the corresponding operator in a corresponding first calculation unit, the first calculation unit being one or more of calculation units of the terminal.

4. The method of claim 1, wherein, The feedback information further carries first reference information. The method further comprises: after obtaining the first operation result, when a difference between first performance information and the first reference information meets a preset condition, sending the first performance information to the server, wherein the first performance information is used to indicate running performance of at least one operator with respect to the corresponding first operation mode, and the first operation result is obtained based on the feedback information.

5. The method of claim 1, wherein, The method further comprises: when the feedback information is not received within the first time, or the feedback information cannot be used to determine the first operation mode of each operator in the neural network model, obtaining a second operation result of to-be-processed data based on a preset second operation mode of each operator in the neural network model.

6. The method of claim 5, wherein, The method further comprises: after obtaining the second operation result, sending second performance information to the server, the second performance information being used to indicate running performance of at least one operator with respect to the corresponding second operation mode.

7. The method of claim 1, wherein, The feedback information contains operator performance information and / or model operation strategy information for the neural network model, the operator performance information comprising information of running performance of a corresponding operator with respect to a specified parameter, the model operation strategy comprising the first operation mode of each operator in the neural network model, the operator performance information indicating running performance of an operator in the neural network model when the operator operates in at least one operation mode, and the operator performance information being able to support determination of the first operation mode of the corresponding operator according to running performance of the corresponding operator.

8. The method according to any one of claims 1 to 7, characterized in that, The neural network model is not associated with hardware of the terminal.

9. A model performance optimization method characterized by, The method applied to a server comprises: receive a first request sent by a terminal, the first request carrying operator information of a neural network model and running environment information of the neural network model; send feedback information to the terminal, the feedback information being obtained based on the operator information and the running environment information, and the feedback information being capable of being used to determine a first operation mode of each operator of a plurality of operators of the neural network model; the terminal has an application program and a target software development kit (SDK) installed therein, the application program corresponding to the neural network model; the receiving of the first request sent by the terminal comprises: receive the first request sent by the target SDK, the first request being generated based on target configuration information and a second request in a case where the target configuration information is included in the target SDK, the second request being sent by the application program to the target SDK, the second request including information of the neural network model, and the target configuration information being used to indicate information content about the neural network model that the terminal is capable of sending to the server.

10. The method of claim 9, wherein, further comprising: receive first performance information sent by the terminal, a difference between the first performance information and first reference information conforming to a preset condition, and the feedback information carrying the first reference information; update corresponding operator performance information in the server based on the first performance information.

11. The method of claim 9, wherein, further comprising: if the server does not obtain the feedback information, store the operator information and the running environment information of the neural network model.

12. The method of claim 11, wherein, further comprising: receive second performance information sent by the terminal, the second performance information being used to indicate running performance of at least one of the operators with respect to a corresponding second operation mode; update the server according to the second performance information, the operator information and the running environment information of the neural network model.

13. A model performance optimization apparatus, characterized by, applied to a terminal, the apparatus comprising: a sending module configured to send a first request to a server, the first request carrying operator information of a neural network model and running environment information of the neural network model; a receiving module configured to receive, within a first time, feedback information sent by the server for the first request, the feedback information being capable of being used to determine a first operation mode of each operator of a plurality of operators of the neural network model; the terminal has an application program and a target software development kit (SDK) installed therein, the application program corresponding to the neural network model; the sending module is configured to: send a second request to the target SDK through the application program, the second request including information of the neural network model; generate the first request based on target configuration information and the second request in a case where the target configuration information is included in the target SDK, and send the first request to the server by the target SDK, the target configuration information being used to indicate information content about the neural network model that the terminal is capable of sending to the server.

14. The apparatus of claim 13, wherein, the feedback information further carries first reference information; the sending module is further configured to: After obtaining the first operation result, the first performance information is sent to the server when a difference between the first performance information and the first reference information meets a preset condition, wherein the first performance information is used to indicate running performance of at least one operator with respect to a corresponding first operation mode, and the first operation result is obtained based on the feedback information.

15. The apparatus of claim 13 or 14, wherein, The device further comprises a processing module; The processing module is configured to: If the feedback information is not received within the first time or the feedback information cannot be used to determine the first operation mode of each operator in the plurality of operators of the neural network model, a second operation result of the to-be-processed data is obtained based on a preset second operation mode of each operator in the plurality of operators of the neural network model.

16. The apparatus of claim 15, wherein, The sending module is further configured to: After obtaining the second operation result, second performance information is sent to the server, wherein the second performance information is used to indicate running performance of at least one operator with respect to a corresponding second operation mode.

17. A model performance optimization apparatus, comprising: The device is applied to a server, and the device comprises: A receiving module configured to receive a first request sent by a terminal, wherein the first request carries operator information of a neural network model and running environment information of the neural network model; A sending module configured to send feedback information to the terminal, wherein the feedback information is obtained based on the operator information and the running environment information, and the feedback information can be used to determine a first operation mode of each operator in a plurality of operators of the neural network model; The terminal is installed with an application program and a target software development kit (SDK), and the application program corresponds to the neural network model; The receiving module is configured to: receive the first request sent by the target SDK, and generate the first request based on target configuration information and a second request, wherein the second request is sent by the application program to the target SDK, the second request carries information of the neural network model, and the target configuration information is used to indicate information content about the neural network model that can be sent by the terminal to the server.

18. The apparatus of claim 17, wherein, The device further comprises an updating module; The receiving module is further configured to receive first performance information sent by the terminal, wherein a difference between the first performance information and first reference information meets a preset condition, and the feedback information carries the first reference information; The updating module is configured to update corresponding operator performance information in the server based on the first performance information.

19. The apparatus of claim 17, wherein, The device further comprises a storage module; The storage module is configured to store the operator information and the running environment information of the neural network model if the server does not obtain the feedback information.

20. The apparatus of claim 19, wherein, The device further comprises an updating module; The receiving module is further configured to receive second performance information sent by the terminal, wherein the second performance information is used to indicate running performance of at least one operator with respect to a corresponding second operation mode; The updating module is configured to update the server according to the second performance information, the operator information and the running environment information of the neural network model.

21. A terminal, characterized by The terminal comprises at least one processor, a memory, and instructions stored on the memory and executable by the at least one processor to implement the steps of the method of any of claims 1-8.

22. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any of claims 1-8.

23. A server, characterized by The server comprises at least one processor, a memory, and instructions stored on the memory and executable by the at least one processor to implement the steps of the method of any of claims 9-12.

24. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any of claims 9-12.

Images