Data processing method, application processor, external chip and electronic device

Abstract

The application discloses a data processing method, an application processor, an external chip and an electronic device, and belongs to the computer field. The method is applied to the application processor and includes the following steps: sending first data and model input data of a first model to an external chip, the first data being part of model data, and the model data being data required for running the first model; receiving a data request sent by the external chip; in response to the data request, sending second data to the external chip, the second data including data other than the first data in the model data; and receiving a model output result sent by the external chip, the model output result being a data processing result corresponding to the model input data.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a data processing method, an application processor, an external chip, and an electronic device. Background Technology

[0002] With the rapid development and application of artificial intelligence technology, more and more models are being deployed on portable electronic devices such as mobile phones. Currently, models deployed on electronic devices mainly run on the application processor (AP). However, due to the AP processor's strong general-purpose capabilities but weak specialized capabilities, the energy efficiency and power consumption of models running on the AP processor are relatively low, resulting in low data processing efficiency when processing data through the model. Summary of the Invention

[0003] The purpose of this application is to provide a data processing method, application processor, external chip, and electronic device that can improve the energy efficiency of the model running on the electronic device, reduce power consumption, and thus improve the efficiency of data processing.

[0004] In a first aspect, embodiments of this application provide a data processing method applied to an application processor, the method comprising:

[0005] Send the first data and model input data of the first model to the external chip. The first data is part of the model data, and the model data is the data required to run the first model.

[0006] Receive data requests sent by external chips;

[0007] In response to a data request, second data is sent to the external chip. The second data includes data from the model data other than the first data.

[0008] It receives the model output results sent by the external chip. The model output results are the data processing results corresponding to the model input data.

[0009] Secondly, embodiments of this application provide a data processing method applied to an external chip, the method comprising:

[0010] Receive and store the first data and model input data of the first model sent by the application processor. The first data includes a portion of the model data, which is the data required to run the first model.

[0011] Load the first data and send a data request to the application processor to obtain the second data in the application processor, the second data including the data in the model data other than the first data;

[0012] Based on the first and second data, process the model input data to obtain the model output results corresponding to the model input data;

[0013] Send the model output results to the application processor.

[0014] Thirdly, embodiments of this application provide an application processor configured to perform the steps of the method described in the first aspect.

[0015] Fourthly, embodiments of this application provide an external chip configured to perform the steps described in the second aspect.

[0016] Fifthly, embodiments of this application provide an electronic device, including an application processor and an external chip;

[0017] The application processor is configured to perform the steps described in the first aspect;

[0018] The external chip is configured to perform the steps described in the second aspect.

[0019] In a sixth aspect, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first or second aspect.

[0020] In a seventh aspect, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the methods described in the first or second aspect.

[0021] Eighthly, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as described in the first or second aspect.

[0022] In this embodiment, the application processor sends first data and model input data to the external chip. The first data is a portion of the model data of the first model. The external chip processes the model input data based on the first data and second data obtained by sending a data request to the application processor, obtaining the model output result of the model input data. The second data includes the data in the model data other than the first data. The application processor receives the model output result sent by the external chip. According to this embodiment, by optimizing the system resource configuration, when processing the model input data through the first model, the external chip runs the first model. Compared with the application processor, the external chip has stronger specialization, higher resource utilization, and lower power consumption. Therefore, it can improve the energy efficiency ratio of model operation and reduce power consumption, thereby improving the efficiency of data processing. Attached Figure Description

[0023] Figure 1 These are schematic diagrams of electronic devices provided in some embodiments of this application;

[0024] Figure 2 These are schematic diagrams illustrating data processing methods provided in some embodiments of this application;

[0025] Figure 3 This is a schematic diagram of the memory of an external chip provided in some embodiments of this application;

[0026] Figure 4 This is a schematic diagram illustrating the division method of the first model provided in some embodiments of this application;

[0027] Figure 5 These are schematic diagrams of electronic devices provided in some embodiments of this application;

[0028] Figure 6 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0030] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0031] Before providing a detailed description of the embodiments of the present invention, related technologies are introduced. As mentioned above, in order to enable models to be deployed on electronic devices, models are typically trimmed, compressed, and quantized to reduce their size and computing power for operation on the electronic device. However, although trimming, compression, and quantization may allow the model to run on the AP processor of an electronic device, the AP processor, due to its versatility across numerous electronic device manufacturers and a wider range of application scenarios, exhibits strong general-purpose capabilities but weak specialized capabilities. This makes it difficult for the AP processor to adequately adapt to the computational requirements of the model, resulting in idle or underutilized hardware resources, while some critical resources may be overloaded, leading to low overall resource utilization and reduced energy efficiency of model operation. Furthermore, to handle various types of tasks, the AP processor requires complex instruction sets and functional units, resulting in many units being active during operation, consuming significant energy and leading to high power consumption. The low energy efficiency of the model prevents the full utilization of the hardware's parallel computing capabilities, resulting in slow data processing speed and low efficiency when processing data through the model.

[0032] In view of this, in order to solve the problem of poor energy efficiency of models running on electronic devices in related technologies, this application provides a data processing method, application processor, external chip, electronic device, storage medium and program product.

[0033] The data processing method, application processor, external chip, electronic device, storage medium, and program product provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0034] The data processing method provided in this application can be applied to scenarios where data processing is performed using any model in an electronic device. It should be noted that the data processing method provided in this application can be executed by an electronic device. This application uses an electronic device executing the data processing method as an example to illustrate the data processing method provided in this application.

[0035] See Figure 1 This is a schematic diagram of the electronic device provided in the embodiments of this application, such as... Figure 1 As shown, the electronic device 100 includes an application processor 101 and an external chip 102 connected by communication. The application processor 101 sends data to the external chip 102 through a first interface 103, and the external chip 102 sends data to the application processor 101 through a second interface 104.

[0036] In some embodiments of this application, the external chip 102 is a dedicated processor for running the model. The external chip 102 is optimized for model running scenarios in terms of hardware configuration and functional design, focusing solely on model running tasks, unlike the application processor 101 which needs to handle multiple different types of workloads. This allows the external chip 102 to concentrate all its resources on completing model running; therefore, compared to the application processor 101, the external chip 102 has higher resource utilization when running models. Furthermore, the external chip 102 retains only the necessary functional units related to model running, eliminating redundant parts; therefore, its power consumption is lower when running models.

[0037] See Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application. This method is applied to, for example... Figure 1 The electronic devices shown, such as Figure 2 As shown, the method includes steps 210-250, which will be explained in detail below.

[0038] Step 210. The application processor sends the first data and model input data of the first model to the external chip. The first data is part of the model data, and the model data is the data required to run the first model.

[0039] In some embodiments of this application, the first model can be any model deployed on an electronic device. Multiple models can be set on the electronic device. In practical applications, users can select the model in the electronic device that needs to be optimized and accelerated based on functional and performance goals, and define the selected model as the first module.

[0040] In some embodiments of this application, for a model downloaded to an electronic device for the first time, the electronic device first stores the model data of the model in the application processor. After the first model is determined, the application processor then transmits the model data of the first model to an external chip through a first interface, so as to optimize and accelerate the first model through a dedicated external chip.

[0041] In some embodiments of this application, the model data of the first model refers to the data required to run the first model. Model data includes, but is not limited to, model parameters, instruction sets, and weight data. Parameters refer to a series of variables learned by the model during training; these are the core components of the model, determining its behavior and performance. Taking a linear regression model as an example, parameters include the intercept and slope. During model operation, these parameters are learned and adjusted using training data, enabling the model to find the optimal relationship between input features and output targets, thereby predicting new data. In more complex deep learning models, parameters can represent the connection strength between neurons, bias terms, etc., which collectively determine the model's ability to extract and map features from input data. The instruction set is a predefined set of rules and operation sequences that define the model's computational logic and process, guiding how the model processes and computes input data to achieve specific tasks or functions. For example, in Recurrent Neural Networks (RNNs) or Long Short-Term Memory Networks (LSTMs) for Natural Language Processing, the instruction set defines how to process the input word vectors at each time step, including how to update the hidden state and how to calculate the output, thereby enabling the modeling and processing of natural language sequences, such as text generation and sentiment analysis. Weight data describes the strength or importance of the associations between different elements in the model. During model training, the weight data is continuously adjusted using optimization methods such as backpropagation, allowing the model to learn the inherent features and patterns of the input data. For example, in an image recognition model, different weight values ​​determine the contribution of each pixel or feature to the final classification result. By adjusting the weight data, the model can accurately identify different image categories. In summary, parameters are the variables learned by the model, the instruction set guides the model's computational process, and weight data, as part of the parameters, plays a crucial role in the model's feature extraction and decision-making processes. Together, they enable the model to complete various complex tasks.

[0042] In some embodiments of this application, although the application processor has sufficient space to store the model data of the first model due to its large data volume, the external chip typically has a smaller capacity, insufficient to store all the model data of the first model, considering the cost of the external chip. Therefore, in this embodiment, the application processor only transmits a portion of the model data of the first model, i.e., the first data, to the external chip. The remaining model data is dynamically transmitted to the external chip during the operation of the first model through data interaction between the application processor and the external chip.

[0043] In some embodiments of this application, the model input data of the first model refers to the data provided to the first model for processing, which forms the basis of the model's output results. The data types of the model input data include, but are not limited to, numerical data, text data, image data, and audio data. Each time the electronic device runs the first model, the application processor receives the externally input model input data, and then the application processor transmits the model input data to an external chip for processing.

[0044] In some embodiments of this application, the application processor can transmit the first data and the model input data to the external chip after receiving the model input data of the first model. This way, when the external chip does not need to process data, data is not sent to the external chip, thus avoiding the occupation of the external chip's memory space and reducing the memory usage on the external chip.

[0045] In other embodiments of this application, the application processor, upon determining the first model and powering on the electronic device, can first send the first data of the first model to the external chip, and then, upon receiving the model input data of the first model, send the model input data to the external chip. In this way, by transmitting the first data to the external chip in advance, when the external chip needs to process the model input data, there is no need to wait for the first data transmission process, greatly reducing data processing time delays and improving data processing efficiency.

[0046] Step 220. The external chip receives and stores the first data and model input data of the first model sent by the application processor.

[0047] In some embodiments of this application, the external chip integrates a memory. Typically, the first model is a pre-trained model; therefore, the model data of the first model is fixed data that does not change as the first model runs, and thus the first data is fixed data. Although the model input data is new data input each time data processing is performed, it does not change during the operation of the first model based on the model input data, that is, during the processing of the model input data by the first model. Therefore, after receiving the first data and model input data of the first model sent by the application processor, the external chip stores the first data and model input data in its own memory. In this way, during the processing of the model input data, the external chip can directly obtain the first data and model input data from its own memory, without dynamically obtaining them from the application processor, thereby improving data processing efficiency and energy efficiency.

[0048] In some embodiments of this application, the external chip uses on-chip memory. On-chip memory offers high data transfer speeds, thus improving data access efficiency and reducing power consumption.

[0049] Step 230. The external chip loads the first data and sends a data request to the application processor to obtain the second data in the application processor. The second data includes the model data other than the first data. In some embodiments of this application, the external chip processes the model input data to obtain the model output result corresponding to the model input data, that is, the data processing result corresponding to the model input data.

[0050] In some embodiments of this application, upon receiving first data and model input data, the external chip processes the model input data based on the model execution order of the first model. When processing the model input data, the external chip loads the data required for processing the current model input data from the first data stored in the external chip, and processes the model input data based on the acquired data. If the first data does not contain the data required for processing the current model input data, a data request is sent to the application processor to obtain the data required for processing the current model input data from the application processor. The model execution order of the first model refers to the order in which the model input data flows and is processed between the various layers or components of the first model during its operation. Taking a neural network model as an example, the model execution order of the first model is the forward propagation order of the network.

[0051] In some embodiments of this application, the data request sent by the external chip is used to instruct the application processor to send the data currently required by the external chip. The data currently required by the external chip can be part or all of the second data.

[0052] Step 240. The application processor receives a data request sent by the external chip.

[0053] Step 250. In response to the data request, the application processor sends second data to the external chip.

[0054] In some embodiments of this application, when the application processor receives a data request sent by the external chip, it determines the data currently required by the external chip based on the data request, thereby obtaining the data currently required by the external chip from the second data and sending the data to the external chip.

[0055] Step 260. The external chip processes the model input data based on the first data and the second data to obtain the model output results corresponding to the model input data.

[0056] In some embodiments of this application, the external chip processes the model input data according to the model execution order of the first model, based on its own stored first data and received second data, until the first model finishes running, and determines that the processing of the model input data is complete. The final output of the first model is then determined as the data processing result of the model input data.

[0057] Step 270. The external chip sends the model output results to the application processor.

[0058] Step 280. The application processor receives the model output results.

[0059] In some embodiments of this application, after the external chip receives the model output result from the model data input, it sends the model output result to the application processor so that the application processor can output the model output result.

[0060] The data processing method provided in this embodiment involves an application processor sending first data and model input data to an external chip. The first data is a portion of the model data of a first model. Based on the first data and second data obtained by sending a data request to the application processor, the external chip processes the model input data to obtain a model output result. The second data consists of other model data besides the first data. The application processor receives the model output result sent by the external chip. According to this embodiment, by optimizing system resource configuration, when processing the model input data through the first model, the external chip runs the first model. Compared to the application processor, the external chip is more specialized, has higher resource utilization, and lower power consumption, thus improving the energy efficiency ratio of model operation and reducing power consumption, thereby improving data processing efficiency.

[0061] Furthermore, during the aforementioned data processing, the external chip dynamically acquires the second data by interacting with the application processor. This eliminates the need for the external chip to store all model data at once, reducing the demand for external storage media and allowing the use of smaller, lower-cost memory, thereby lowering the overall cost of the external chip.

[0062] In some embodiments, the parameter and instruction set of the first model forms the basis for running the first model. When using the first model for data processing, the parameter and instruction set needs to be loaded all at once. Considering that the parameter and instruction set occupies a relatively small amount of space, the parameter and instruction set of the first model can be stored as the first data on an external chip. In this way, the external chip can load the parameter and instruction set all at once, without dynamically retrieving the parameter and instruction set from the application processor, thereby improving data processing efficiency.

[0063] In some embodiments, the total weight data of the first model is large, while the memory capacity of the external chip is small due to cost considerations. Therefore, it is difficult for the external chip to store all the weight data of the first model at once. Based on this, the total weight data of the first model can be divided into first weight data and second weight data. Only the first weight data is transmitted to the external chip as the first data, while the second weight data is dynamically transmitted to the external chip as the second data. The first weight data is a portion of the weight data in the model data, and the second weight data is the weight data in the model data excluding the first weight data. Because the first weight data is transmitted to the external chip by the application processor before data processing and stored in the external chip's memory, the external chip does not need to dynamically retrieve the first weight data from the application processor during the processing of the model input data. Therefore, the first weight data can be considered static weight data that does not require dynamic retrieval. The second weight data, however, needs to be dynamically retrieved by the external chip from the application processor during data processing. Therefore, the second weight data can be considered dynamic weight data that requires dynamic retrieval.

[0064] By dividing the weight data of the first model into first weight data and second weight data, only the first weight data is designated as the first data and stored in the external chip, while the second weight data is dynamically retrieved from the application processor by the external chip during the operation of the first model. This approach ensures that the performance of the first model is not affected while maintaining the cost of the external chip.

[0065] In some embodiments, considering that the first model typically includes multiple layers or components, for example, taking a neural network model as an example, it includes an input layer, convolutional layer, pooling layer, output layer, etc. Different layers or components use different weight values. During the process of processing the model input data through the first model, the external chip dynamically loads the required weight data according to the model execution order. Based on this, the first model can be pre-divided into at least one first module and at least one second module according to the model execution order. The weight data applied to at least one first module is determined as the first weight data, and the weight data applied to at least one second module is determined as the second weight data. Dividing the weight data in this way ensures that the divided weight data corresponds to the modules of the first model. Thus, during the operation of the first model, weight data is loaded on a module-by-module basis, thereby reducing the data transmission frequency and improving data processing efficiency.

[0066] In some embodiments, the size and usage of data such as parameters, instruction sets, weight data, and model input data are different. Therefore, in order to better utilize the storage space of the external chip's memory, the external chip's memory can be divided into multiple storage areas, with different storage areas used to store different data.

[0067] In some embodiments of this application, the first data includes parameters, an instruction set, and first weight data, and the second data includes second weight data. Based on this, as... Figure 3 As shown, the memory 300 of the external chip includes a first storage area 301, a second storage area 302, a third storage area 303, and a fourth storage area 304. The first storage area 301 stores parameters and instruction sets; the second storage area 302 stores model input data; the third storage area 303 stores first weight data; and the fourth storage area 304 stores second weight data dynamically obtained from the application processor. Based on this, after receiving the first data, the external chip stores the parameters and instruction sets in the first storage area 301 and the first weight data in the third storage area 303. After receiving the model input data, the external chip stores the model input data in the second storage area 302. After obtaining the second data from the application processor, the external chip stores the second data in the fourth storage area 304.

[0068] In some embodiments of this application, it is considered that the first model will generate some intermediate variables for subsequent use during its operation. Intermediate variables refer to various data representations or features between the model input data and the model output results. For example, when the model input data is image data and the first model is an image processing model, some feature maps will be generated during the processing of the model input data by the first model; these feature maps are intermediate variables. These intermediate variables are still needed during the operation of the first model, therefore, as... Figure 3 As shown, the memory 300 may also include a fifth storage area 305 for storing intermediate variables. Therefore, during the execution of the first model, the external chip stores the generated intermediate variables in the fifth storage area 305 for subsequent use.

[0069] In some embodiments of this application, the capacities of the first, second, third, and fifth storage areas are all greater than or equal to the amount of data to be stored. For example, if the data size of the parameters and instruction set of the first model is 3MB, then the capacity of the first storage area is greater than or equal to 3MB. If the data size of the model input data of the first model is 128kB, then the capacity of the second storage area is greater than or equal to 128kB. If the data size of the first weight data of the first model is 900MB, then the capacity of the third storage area is greater than or equal to 900MB. If the data size of the intermediate variables generated during the operation of the first model is approximately 60MB, then the capacity of the fifth storage area is greater than or equal to 60MB. The data sizes of the parameters and instruction set, the model input data, and the intermediate variables generated during the operation of the first model can be obtained in advance through analysis of the first model.

[0070] In some embodiments of this application, since the external chip dynamically obtains the second data from the application processor during the processing of model input data, it only needs to obtain the data currently required, without needing to obtain all the second data at once. Therefore, the capacity of the fourth storage area can be less than the total amount of the second data. Based on this, the capacity of the fourth storage area can be determined based on the total capacity of the memory, the capacities of the first, second, third, and fifth storage areas, as long as the total capacity of the first, second, third, fourth, and fifth storage areas does not exceed the total capacity of the memory. For example, if the total memory capacity is 1024MB, the capacity of the first storage area is 3MB, the capacity of the second storage area is 128kB, the capacity of the third storage area is 900MB, and the capacity of the fifth storage area is 60MB, then the capacity of the fourth storage area can be any capacity less than or equal to 62336kB.

[0071] In some embodiments of this application, considering that the smaller the capacity of the fourth storage area, the smaller the amount of second data the external chip can obtain from the application processor each time, and the more times it needs to interact with the application processor to obtain all the second data, the larger the capacity of the fourth storage area should be to reduce the number of interactions between the external chip and the application processor. For example, when the total memory capacity is 1024MB, the capacity of the first storage area is 3MB, the capacity of the second storage area is 128kB, the capacity of the third storage area is 900MB, and the capacity of the fifth storage area is 60MB, the capacity of the fourth storage area can be set to 60MB.

[0072] In some embodiments, to improve data processing efficiency, the first module and the second module are alternately distributed according to the model execution order of the first model. For example, see... Figure 4 The first model is divided into 14 modules according to the execution order of the first model. Figure 4 The system uses numbers 1-14 to distinguish 14 modules. Modules numbered 1, 3, 5, 7, 9, 11, and 13 are designated as the first modules, and modules numbered 2, 4, 6, 8, 10, 12, and 14 are designated as the second modules. This allows the external chip to retrieve the weight data for the next second module from the application processor while running the first module. For example, while running the first module (number 1), it retrieves the weight data for the second module (number 2); while running the first module (number 3), it retrieves the weight data for the second module (number 4), and so on, thereby improving data processing efficiency.

[0073] In some embodiments, to completely hide the transmission time of the second data during the operation of the first module, the first module and the second module can be divided as follows:

[0074] Based on the capacity of the fourth storage area and the bandwidth of the first interface, the duration of a single transmission between the application processor and the external chip is determined.

[0075] According to the execution order of the first model, the first model is divided by alternating the single transmission duration and the application processor capacity to obtain at least one first module based on the single transmission duration and at least one second module based on the capacity of the fourth storage area. The runtime of the first module is greater than or equal to the single transmission duration, and the weight data volume of the second module is less than or equal to the capacity of the fourth storage area.

[0076] In some embodiments of this application, the bandwidth of the first interface is used to indicate the amount of data transmitted by the first interface per unit time. The duration of a single transmission between the application processor and the external chip is the ratio of the capacity of the fourth storage area to the bandwidth of the first interface. The duration of a single transmission is used to indicate the time required to transmit second data, the amount of data of which is the same as the capacity of the fourth storage area, through the first interface. Since both the bandwidth of the first interface and the capacity of the fourth storage area are fixed values, the duration of a single transmission is also a fixed value.

[0077] In some embodiments of this application, the amount of weight data for each second module, divided based on the capacity of the fourth storage area, is less than or equal to the capacity of the fourth storage area. This ensures that the fourth storage area is sufficient to store all the weight data required by any second module. Since the amount of weight data for each second module is less than or equal to the capacity of the fourth storage area, the time required to transmit the weight data for any second module is necessarily less than or equal to the duration of a single transmission. Therefore, when dividing the first modules based on the duration of a single transmission, ensuring that the runtime of each first module is greater than or equal to the duration of a single transmission guarantees that during the operation of the first module, the external chip can acquire all the weight data for the next second module to be run.

[0078] By dividing the first model according to the single transmission duration and the capacity of the fourth storage area in the above manner, the runtime of the first module is greater than or equal to the transmission duration required to transmit the weight data of the next second module to be run. In this way, during the operation of the first module, the external chip can obtain all the weight data of the next second module to be run. After the first module is completed, the next second module to be run can be run directly without waiting for data transmission, thus improving data processing efficiency.

[0079] Accordingly, based on the above module division method, the external chip can process the model input data through the following steps:

[0080] Load the set of parameters and instructions from the first data;

[0081] When the model input data is processed through the first module, the first module is run based on the weight data applied to the first module in the first weight data;

[0082] During the execution of the first module, a data request is sent to the application processor to obtain the weight data of the next second module to be executed, which is sent by the application processor.

[0083] The weight data of the next second module to be run is stored in the fourth storage area;

[0084] When the model input data is processed through the second module, the second module is run based on the weight data stored in the fourth storage area;

[0085] After all the first and second modules have completed their runs, the output of the last module is determined as the model output result corresponding to the model input data.

[0086] For example, with Figure 4 For example, during the operation of the first module (numbered 1), the external chip sends a data request to the application processor to obtain the weight data of the second module (numbered 2). In response to this request, the application processor transmits the weight data corresponding to the second module (numbered 2) from the second data to the external chip via the first interface. The external chip then stores the weight data of the second module (numbered 2) in the fourth storage area. After the first module (numbered 1) finishes running, the external chip loads the weight data of the second module (numbered 2) based on the address in the fourth storage area, and then runs the second module (numbered 2) based on this weight data. When the second module (numbered 2) finishes running, the external chip loads the weight data of the first module (numbered 3) from the third storage area, and runs the first module (numbered 3) based on this weight data. Simultaneously, during the running of the first module (numbered 3), it sends a data request to the application processor to obtain the weight data of the second module (numbered 4). The application processor responds to this data request by transferring the weight data corresponding to the second module (numbered 4) from the second data to the external chip via the first interface. The external chip then stores the weight data of the second module (numbered 4) in the fourth storage area, effectively replacing the data stored in the fourth storage area with the weight data of the second module (numbered 4). After the first module (numbered 3) finishes running, the external chip loads the weight data of the second module (numbered 4) based on the address in the fourth storage area, and runs the second module (numbered 4) based on this weight data. This process is repeated until all the first and second modules of the first model have been run, and the output of the last module is used as the model output.

[0087] The above approach ensures that the transmission time of the second data is completely hidden during the operation of the first model, thus avoiding additional time waste and improving overall performance.

[0088] Based on the data processing method provided in the above embodiments, this application also provides an application processor and an external chip. The application processor provided in this embodiment is configured to implement... Figures 2 to 4 In the method embodiments, the various processes implemented by the application processor are described. The external chip provided in this embodiment is configured to implement... Figures 2 to 4 The various processes implemented by the external chip in the method embodiment will not be described again here to avoid repetition.

[0089] The application processor and external chip in the embodiments of this application are components in an electronic device, such as integrated circuits or chips. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. The embodiments of this application do not specifically limit the scope.

[0090] The application processor and external chip in the embodiments of this application can be devices with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application does not specifically limit the specific operating system used.

[0091] Optionally, such as Figure 5 As shown, this application embodiment also provides an electronic device 500, including a processor 501 and a memory 502. The memory 502 stores a program or instructions that can run on the processor 501. When the program or instructions are executed by the processor 501, they implement the various steps of the above-described data processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0092] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0093] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0094] The electronic device 600 includes, but is not limited to, components such as: radio frequency unit 601, network module 602, audio output unit 603, input unit 604, sensor 605, display unit 606, user input unit 607, interface unit 608, memory 609, and processor 610.

[0095] Those skilled in the art will understand that the electronic device 600 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 610 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0096] In some embodiments, the processor 610 includes an application processor for sending first data and model input data for a first model to an external chip, wherein the first data is a portion of the model data and the model data is the data required to run the first model;

[0097] Receive data requests sent by external chips;

[0098] In response to a data request, second data is sent to the external chip. The second data includes data from the model data other than the first data.

[0099] It receives the model output results sent by the external chip. The model output results are the data processing results corresponding to the model input data.

[0100] In some embodiments, the processor 610 includes an external chip for receiving and storing first data and model input data of the first model sent by the application processor. The first data includes a portion of the model data, which is the data required to run the first model.

[0101] Load the first data and send a data request to the application processor to obtain the second data in the application processor, the second data including the data in the model data other than the first data;

[0102] Based on the first and second data, process the model input data to obtain the model output results corresponding to the model input data;

[0103] Send the model output results to the application processor.

[0104] In some embodiments, the first data includes parameters, an instruction set, and first weight data; the second data includes second weight data, which includes weight data in the model data other than the first weight data; the processor 610 includes an external chip, which includes a first storage area, a second storage area, and a third storage area; the processor 610 is further configured to:

[0105] After receiving the first data and model input data of the first model sent by the application processor, the parameters and instruction set are stored in the first storage area;

[0106] The model input data is stored in the second storage area;

[0107] The first weighted data is stored in the third storage area.

[0108] In some embodiments, the first weight data includes weight data of at least one first module in the first model, the second weight data includes weight data applied to at least one second module in the first model, and the external chip further includes a fourth storage area.

[0109] The processor 610 includes an external chip, specifically used for:

[0110] Load the set of parameters and instructions from the first data;

[0111] During the execution of the first module, a data request is sent to the application processor to obtain the weight data of the next second module to be executed, which is sent by the application processor.

[0112] The weight data of the next second module to be run is stored in the fourth storage area;

[0113] When the model input data is processed through the first module, the first module is run based on the weight data applied to the first module in the first weight data;

[0114] When the model input data is processed through the second module, the second module is run based on the weight data stored in the fourth storage area;

[0115] After all the first and second modules have completed their runs, the output of the last module is determined as the model output result corresponding to the model input data.

[0116] In some embodiments, the processor 610 includes an external chip, the external chip including an intermediate variable storage area, and the processor 610 is further configured to:

[0117] During the process of processing the model input data, the intermediate variables generated by the first model are stored in the intermediate variable storage area.

[0118] It should be understood that, in this embodiment, the input unit 604 may include a graphics processing unit (GPU) 6041 and a microphone 6042. The GPU 6041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 607 includes at least one of a touch panel 6071 and other input devices 6072. The touch panel 6071 is also called a touch screen. The touch panel 6071 may include two parts: a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0119] The memory 609 can be used to store software programs and various data. The memory 609 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 609 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 609 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0120] Processor 610 may include one or more processing units; optionally, processor 610 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 610.

[0121] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0122] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0123] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above data processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0124] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0125] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the data processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0126] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0128] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data processing method, characterized in that, Applied to an application processor, the method includes: Send the first data and model input data of the first model to the external chip. The first data is part of the model data, and the model data is the data required to run the first model. Receive data requests sent by the external chip; In response to the data request, second data is sent to the external chip, the second data including data in the model data other than the first data; The system receives the model output result sent by the external chip, where the model output result is the data processing result corresponding to the model input data.

2. The method according to claim 1, characterized in that, The first data includes parameters, instruction set and first weight data, the second data includes second weight data, the first weight data includes some weight data in the model data, and the second weight data includes weight data in the model data other than the first weight data.

3. The method according to claim 2, characterized in that, The first model includes at least one first module and at least one second module, divided according to the execution order of the model; The first weight data includes the weight data applied to the first module in the model data; The second weight data includes the weight data applied to the second module in the model data.

4. The method according to claim 3, characterized in that, The first module and the second module are alternately distributed according to the execution order of the first model.

5. The method according to claim 3, characterized in that, The application processor sends data to the external chip through a first interface, and the external chip includes a storage area for storing the second data. The division methods between the first module and the second module include: Based on the capacity of the storage area storing the second data and the bandwidth of the first interface, the duration of a single transmission between the application processor and the external chip is determined; According to the execution order of the first model, the first model is divided by alternating the single transmission duration and the capacity of the application processor to obtain at least one first module divided based on the single transmission duration and at least one second module divided based on the capacity of the storage area storing the second data. The runtime of the first module is greater than or equal to the single transmission duration, and the weighted data volume of the second module is less than or equal to the capacity of the storage area storing the second data.

6. A data processing method, characterized in that, Applied to external chips, the method includes: Receive and store first data and model input data of a first model sent by the application processor, wherein the first data includes a portion of the model data, and the model data is the data required to run the first model; Load the first data and send a data request to the application processor to obtain the second data in the application processor, the second data including data in the model data other than the first data; Based on the first data and the second data, the model input data is processed to obtain the model output result corresponding to the model input data; The model output is sent to the application processor.

7. The method according to claim 6, characterized in that, The first data includes parameters, instruction set and first weight data; the second data includes second weight data, which includes weight data other than the first weight data in the model data; the external chip includes a first storage area, a second storage area and a third storage area. After receiving the first data and model input data of the first model sent by the application processor, the method further includes: Store the parameters and the instruction set in the first storage area; The model input data is stored in the second storage area; The first weighted data is stored in the third storage area.

8. The method according to claim 7, characterized in that, The first weight data includes weight data of at least one first module in the first model, the second weight data includes weight data applied to at least one second module in the first model, and the external chip further includes a fourth storage area; The loading of the first data and the sending of a data request to the application processor to obtain the second data in the application processor include: Load the parameters and instruction set from the first data; During the execution of the first module, a data request is sent to the application processor to obtain the weight data of the next second module to be executed, sent by the application processor. The weight data of the next second module to be run is stored in the fourth storage area; The step of processing the model input data based on the first data and the second data to obtain the model output result corresponding to the model input data includes: When the model input data is processed by the first module, the first module is run based on the weight data applied to the first module in the first weight data; When the model input data is processed by the second module, the second module is run based on the weight data stored in the fourth storage area; After all the first modules and all the second modules have completed their operation, the output of the last module is determined as the model output result corresponding to the model input data.

9. The method according to any one of claims 6-8, characterized in that, The external chip includes an intermediate variable storage area, and the method further includes: During the processing of the model input data, the intermediate variables generated by the first model are stored in the intermediate variable storage area.

10. An application processor, characterized in that, The application processor is configured to perform the data processing method as described in any one of claims 1-5.

11. An external chip, characterized in that, The external chip is configured to perform the data processing method as described in any one of claims 6-9.

12. An electronic device, characterized in that, include: Application processors and add-on chips; The application processor is configured to perform the data processing method as described in any one of claims 1-5; The external chip is configured to perform the data processing method as described in any one of claims 6-9.

13. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the data processing method as described in any one of claims 1-5 or 6-9.

14. A computer program product, characterized in that, The program product is stored in a storage medium and is executed by at least one processor to implement the steps of the data processing method as described in any one of claims 1-5 or 6-9.

Images