A reasoning task execution method and related device

Abstract

Embodiments of the present application disclose a reasoning task execution method. In the method, scene awareness can be accurately realized according to application scenario information of one or more aspects of an electronic device, and it is evaluated whether it is an AI scene indicating an intention to enable an AI model, so that the preloading of the AI model is realized before the target application in the electronic device explicitly indicates the function of enabling the AI model in the case of determining that the application scenario corresponding to the electronic device is the AI scene. Moreover, the AI model preloading scheme using the method can directly enable the AI model located in the second storage space to execute the reasoning task after the target application indicates the function of enabling the AI model, reduces the time-consuming of AI model loading, effectively shortens the corresponding reasoning time, efficiently improves the latency performance, and avoids complex software and hardware scheduling logic and the problem of reduced model accuracy caused by model compression technologies such as static pruning.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a reasoning task execution method and related equipment. Background Technology

[0002] With the continuous growth in the scale of artificial intelligence (AI) models, such as large language models, the application of AI models in edge scenarios such as terminal devices faces severe challenges. Specifically, when applying AI models, it is usually necessary to load the AI ​​model from storage devices such as non-volatile memory (NVM) to storage devices such as dynamic random access memory (DRAM) and double data rate synchronous dynamic random access memory (DDR) so that it can be invoked by processes.

[0003] However, the current growth rate of AI model size significantly outpaces the increase in transmission speed brought about by hardware upgrades. This severely impacts latency performance metrics such as the first output token response time (TTFT) for AI models like large language models. For instance, with current NVM read speeds less than 10 GBps, models exceeding 10 bytes will incur at least a 0.5-second read latency. Consequently, loading AI models from NVM to perform inference tasks is inefficient in scenarios such as terminal devices. Summary of the Invention

[0004] This application provides a method for executing inference tasks, which can solve the problem of low execution efficiency when loading AI models from NVM to perform inference tasks in scenarios such as terminal devices. This application also provides corresponding apparatus, devices, computer-readable storage media, and computer program products.

[0005] The first aspect of this application provides a method for performing inference tasks. The method includes: acquiring application scenario information about an electronic device; if, based on the application scenario information, it is determined that the electronic device is in an artificial intelligence (AI) scenario, loading an AI model from a first storage space of the electronic device to a second storage space of the electronic device, wherein the AI ​​scenario indicates an intention to enable the AI ​​model, and the second storage space is used by the target application to enable the AI ​​model's function to perform inference tasks; the first storage space and the second storage space are different; acquiring a trigger instruction, which is either a user-input instruction or an instruction generated by the target application, and the trigger instruction instructs the target application to enable the AI ​​model's function; and, based on the trigger instruction, running the AI ​​model loaded into the second storage space to enable the AI ​​model's function to perform the inference task.

[0006] In the first aspect, scene perception can be achieved more accurately based on one or more application scenario information, such as interaction information between electronic devices and other devices, application information related to the target application, and sensor data of electronic devices. This allows for the estimation of user intent and assessment of whether the scenario indicates an intention to enable an AI model. If an AI scenario is identified, the AI ​​model can be pre-loaded before the target application in the electronic device explicitly instructs to enable the AI ​​model based on a trigger command. This pre-loading approach reduces the loading time of the AI ​​model after the target application instructs to enable it, effectively shortening the corresponding inference time and significantly improving latency performance such as TTFT. Furthermore, this method avoids complex hardware and software scheduling logic and the model accuracy reduction issues caused by model compression techniques such as static and dynamic pruning. The scheduling logic is simple and highly executable.

[0007] In one possible implementation of the first aspect, the time of obtaining the trigger instruction is later than the time of starting to load the AI ​​model from the first storage space to the second storage space.

[0008] In this possible implementation, the intent to enable the AI ​​model can be anticipated based on application scenario information before the target application explicitly instructs to do so, thereby triggering the preloading of the AI ​​model. In this way, when the target application in the electronic device explicitly instructs to enable the AI ​​model, the pre-loaded AI model in a secondary storage space such as memory can be efficiently activated to quickly launch the AI ​​model's functionality and perform the corresponding inference task.

[0009] In one possible implementation of the first aspect, the first storage space is located in a non-volatile memory in the electronic device, and the second storage space is located in the memory of the electronic device.

[0010] In this possible implementation, the second storage space is located in the electronic device's memory. Before the target application explicitly indicates that the AI ​​model needs to be enabled, and if the scenario is inferred to be an AI scenario based on application scenario information, the AI ​​model can be preloaded. In this way, when the target application explicitly indicates that the AI ​​model needs to be enabled, the process or thread of the target application running in memory can efficiently perform the corresponding inference task through the AI ​​model that has been preloaded into memory (i.e., the second storage space).

[0011] In one possible implementation of the first aspect, the application scenario information includes one or more of the following: interaction information between the electronic device and other devices, application information related to the target application, and sensor data of the electronic device.

[0012] In this possible implementation, for example, the interaction information may include timing information of the interaction between the electronic device and other devices, user information of the electronic device and other devices, spatial information of the electronic device and other devices, etc.; the application information may include information on the application call chain involved in the target application using the AI ​​model, such as the usage timing information of the user-mode application, the wake-up timing information of the kernel-mode thread, etc.; and the sensor data may be used to describe the environmental conditions of the electronic device.

[0013] In this possible implementation, the application scenario information may include, but is not limited to, one or more of the following: interaction information between electronic devices and other devices, application information related to the target application, and sensor data of electronic devices. This information can describe the current application scenario from one or more aspects, which helps to accurately identify the corresponding scenario and user intent, and provides a richer data foundation for subsequent data processing.

[0014] In one possible implementation of the first aspect, the method further includes: within a specified range of communication resources, determining the communication resources between the first storage space and the second storage space based on the resource usage and / or energy consumption requirements of the electronic device; and, when it is determined based on application scenario information that the electronic device is in an artificial intelligence (AI) scenario, loading the AI ​​model from the first storage space of the electronic device to the second storage space of the electronic device, including: when it is determined based on application scenario information that the electronic device is in an AI scenario, loading the AI ​​model from the first storage space to the second storage space through communication resources.

[0015] In this possible implementation, based on the number of parameters of the AI ​​model and within the communication resource range between the first and second storage spaces, the input / output (I / O) bandwidth that meets preset energy efficiency requirements can be determined as the communication resource between the first and second storage spaces, according to factors such as the resource usage of the electronic device. This allows the AI ​​model to be loaded with the desired energy efficiency. Therefore, this possible implementation can dynamically control the bandwidth and other communication resources for loading the AI ​​model, enabling the AI ​​model to be loaded under ideal energy efficiency and / or bandwidth conditions, thereby reducing power consumption during the AI ​​model loading process, saving electronic device resources, and ensuring the operating performance of the electronic device.

[0016] In one possible implementation of the first aspect, the method further includes: determining whether the electronic device is in an AI scenario based on application scenario information, preset rules, and / or a machine learning model.

[0017] A second aspect of this application provides a reasoning task execution apparatus, which has the function of implementing the method described in the first aspect or any possible implementation of the first aspect. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function, such as an acquisition module and a processing module.

[0018] A third aspect of this application provides an electronic device including at least one processor, a memory, and computer-executable instructions stored in the memory and executable on the processor. When the computer-executable instructions are executed by the processor, the processor performs a method as described in the first aspect or any possible implementation thereof.

[0019] The fourth aspect of this application provides a computer-readable storage medium storing one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor performs a method as described in the first aspect or any possible implementation thereof.

[0020] The fifth aspect of this application provides a computer program product that stores one or more computer-executable instructions, wherein when the computer-executable instructions are executed by a processor, the processor executes a method as described in the first aspect or any possible implementation thereof.

[0021] A sixth aspect of this application provides a chip system including a processor for supporting an electronic device in implementing the functions described in the first aspect or any possible implementation thereof. In one possible design, the chip system may further include a memory for storing necessary program instructions and data for the electronic device. This chip system may be composed of chips or may include chips and other discrete devices.

[0022] The technical effects of the second to sixth aspects or any of their possible implementations can be found in the first aspect or the technical effects of its related possible implementations, and will not be repeated here. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of an exemplary system architecture provided in an embodiment of this application;

[0024] Figure 2 This is an exemplary schematic diagram of the reasoning task execution method provided in the embodiments of this application;

[0025] Figure 3 This is a schematic diagram of an exemplary processing flow provided in an embodiment of this application;

[0026] Figure 4a This is an exemplary schematic diagram illustrating the time delay of the AI ​​model preloading scheme provided in this application embodiment compared to traditional schemes;

[0027] Figure 4b This is another exemplary schematic diagram illustrating the time delay of the AI ​​model preloading scheme provided in this application embodiment compared to traditional schemes;

[0028] Figure 5 This is a schematic diagram of an embodiment of the reasoning task execution device provided in this application;

[0029] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0031] As will be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0032] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to those processes, methods, products, or apparatus.

[0033] The technical terms used in this application will be introduced below.

[0034] 1. Large Language Model (LLM)

[0035] Large language models are deep learning models trained on massive amounts of text data that can generate natural language text or understand the meaning of language text. They offer a deep understanding of text meaning and can handle various natural language tasks, such as text classification, question answering, and dialogue, making them an important pathway to artificial intelligence.

[0036] Specifically, large language models are a technology that has emerged in recent years. Because large language models undergo meticulous data engineering and training processes, their parameters have learned a wealth of existing natural language processing knowledge. This knowledge can now replace humans in many language-related tasks, such as having large language models write code or perform text summarization.

[0037] As the scale of AI models, such as large language models, continues to grow, their application in edge-side scenarios, including terminal devices, faces severe challenges. Specifically, when applying AI models, they typically need to be loaded from non-volatile memory (NVM) to storage devices such as dynamic random access memory (DRAM) or double data rate synchronous dynamic random access memory (DDR) for process access. However, currently, the growth rate of AI models significantly outpaces the increase in transmission rates brought about by hardware updates, severely impacting latency performance metrics such as the first output token response time (TTFT) for AI models like large language models. For example, with current NVM read speeds less than 10 GBps, models exceeding 10 bytes will experience at least a 0.5-second read latency.

[0038] Furthermore, this model loading process also results in significant power consumption, leading to substantial overall power consumption for terminal devices and other equipment in edge scenarios.

[0039] Currently, a common method to reduce latency performance metrics such as TTFT in AI models is to reduce the number of parameters and compress the model size through model quantization and / or static pruning techniques. Model quantization converts high-precision values ​​(such as floating-point numbers) into low-precision values ​​(such as integers or fixed-point numbers) in the AI ​​model, reducing the amount of data in the parameters and achieving model compression. This makes the converted AI model more efficient in terms of computation and storage. Static pruning techniques, such as weight pruning, reduce computational complexity and decrease the AI ​​model's memory footprint.

[0040] However, this approach can lead to a decrease in model precision, which in turn affects the accuracy of the inference results, resulting in a reduction in the overall accuracy of the inference.

[0041] Another common method to reduce latency performance metrics such as TTFT in AI models is to employ heterogeneous computing power scheduling during the decoding phase of the inference process. This fully utilizes the heterogeneous computing power of the distributed system to mask data transmission operations, thereby achieving efficient inference. However, this approach involves complex hardware and software scheduling logic and is difficult to implement.

[0042] Based on this, the present application provides a reasoning task execution method, which can improve execution efficiency when using an AI model to perform reasoning tasks. Furthermore, in the solution of the present application, the accuracy of the AI ​​model itself does not need to be adjusted, which can better guarantee the performance of the AI ​​model. Moreover, the scheduling logic is relatively simple and highly executable.

[0043] The embodiments of this application can be applied to electronic devices. For example, the electronic device can be a terminal device, a single server or a server cluster, or a virtual machine (VM) or a container (Docker), etc.

[0044] The type of terminal device is not limited here. For example, the terminal device can be a mobile phone, tablet, computer with wireless transceiver capabilities, virtual reality (VR) terminal, augmented reality (AR) terminal, terminal in industrial control, terminal in self-driving, terminal in remote medical care, terminal in smart grid, terminal in transportation safety, terminal in smart city, terminal in smart home, terminal in Internet of Things (IoT), wearable device, robot, etc., or a combination of one or more of these.

[0045] refer to Figure 1 As shown in the example, in this embodiment of the application, the electronic device may have a target application installed. This target application is an application capable of using the functions of an AI model to perform inference tasks. The specific type and function of the target application can vary. For example, the target application can be a system application in the electronic device or a third-party application installed on the electronic device. The target application can be a standalone application or in the form of a mini-program, plugin, or other software module. This embodiment of the application does not limit the specific type and function of the target application.

[0046] The specific type and structure of the AI ​​model are not limited here. For example, the AI ​​model can be a convolutional neural network (CNN), a feedforward neural network (FNN), a recurrent neural network (RNN), a transformer model, a large language model (LLM), etc.

[0047] Furthermore, in this embodiment of the application, the reasoning task execution method can be implemented by a designated application. The designated application can be an application different from the target application, or it can be considered as a judgment model or judgment service, or it can be a plug-in, tool or other form of software module in the target application. In this embodiment of the application, the specific form of the designated application is not limited.

[0048] For example, such as Figure 1 In the example shown, the specified application can be considered as a judgment model. This judgment model can preload the AI ​​model when it is determined that the electronic device is in an AI scenario based on the application scenario information, so that the target application can efficiently enable the preloaded AI model when it needs to enable the AI ​​model function.

[0049] Specifically, based on the aforementioned electronic devices, refer to Figure 2 As shown in the example, the inference task execution method of this application embodiment includes steps 201-205.

[0050] Step 201: Obtain information about the application scenarios of electronic devices.

[0051] In this embodiment of the application, the application scenario information is used to describe the application scenario of the electronic device within a specified time period. For example, it can describe the usage of one or more applications such as the target application in the electronic device, the environment in which the electronic device is located, or the information interaction between the electronic device and other devices.

[0052] Specifically, in some embodiments, the application scenario information includes at least one of the following: interaction information between the electronic device and other devices, application information related to the target application, and sensor data of the electronic device.

[0053] Below, we will provide examples of various possible contents in the application scenario information.

[0054] 1. Interaction information between electronic devices and other devices

[0055] In this example, the other device can be a different electronic device from the electronic device. The device type of the other device can be the same as or different from that of the electronic device. Furthermore, the specific number of the other device can be one; for example, the electronic device and the other devices can be different terminal devices. Alternatively, the specific number of the other devices can be multiple; for example, the electronic device and the other devices can be different computing nodes located in the same distributed system. Therefore, the specific system architecture involved in the electronic device and the other devices can have various forms, and this embodiment does not limit the specific architecture.

[0056] The interaction information between electronic devices and other devices can also be called multi-device data or multi-device collaborative data, used to describe the information exchange and other operations between electronic devices and other devices. There can be various interaction methods between electronic devices and other devices, which are not limited here. For example, the electronic device can communicate with other devices through near field communication (NFC) technology, or through other wireless or wired communication methods.

[0057] The interactive information may include the specific content of the communication between the electronic device and other devices, or it may include one or more of the following: user information of the electronic device and other devices, spatial information of the electronic device and other devices, communication time, number of communications, communication type, etc., to describe the attributes and / or communication methods of the communication operation between the electronic device and other devices.

[0058] For example, the interaction information may include timing information of the interaction between the electronic device and other devices and spatial information of multiple devices (electronic device and other devices); wherein, the timing information may include data sequences collected in chronological order, describing the communication operations between the electronic device and other devices at different time points, and the spatial information describes the respective locations of the electronic device and other devices and / or the spatial relationships between the electronic device and other devices, etc.

[0059] For example, in Figure 3 In the example shown, the interaction information may include user information of multiple devices such as electronic devices and other devices, as well as timing information and spatial information of the interaction between the multiple devices. The timing information, such as the data sequence collected in chronological order, describes the communication operations between electronic devices and other devices at different time points, and the spatial information describes the spatial relationship between electronic devices and other devices.

[0060] In this example, considering that the AI ​​model's functionality is typically activated in specific scenarios and may involve multi-device interaction (for example, when the electronic device is a user's mobile phone, the AI ​​model's functionality may need to be activated when other smart devices such as the user's smart TV, smart car, or laptop are nearby based on communication interactions such as Bluetooth, WiFi, and NFC), furthermore, Figure 3 In the example shown, the interaction data between the multiple devices can also reflect one or more scenarios such as travel, gaming, office work, education, and sports, and can help identify user intent in the above scenarios.

[0061] Therefore, this interactive information can be obtained to perceive the specific situation of multi-terminal collaboration, providing a data foundation for subsequent AI scene judgment, which helps to accurately judge the AI ​​scene in subsequent scene perception and scene intent recognition.

[0062] There are various ways to obtain this interaction information, and no specific method is limited here. For example, if the electronic device and other devices are located in the same distributed system, the interaction information can be obtained based on that distributed system. For instance, the distributed system may include a management node used to manage communication between the electronic device and other devices; the electronic device can then obtain the interaction information from the management node. Alternatively, existing or newly added tools or modules within the electronic device can be used to collect interaction information between the electronic device and other devices based on the electronic device's communication interface, and record it in the electronic device's logs or other files. The electronic device can then extract the interaction information between itself and other devices from the corresponding files.

[0063] It is understandable that the interaction information can be the interaction information between electronic devices and other devices within a specified time period, so as to more accurately reflect the current interaction status of electronic devices and avoid the data volume of the interaction information being too large.

[0064] 2. Application information related to the target application

[0065] In this example, application information related to the target application may include information about one or more applications involved when the electronic device enables the AI ​​model function.

[0066] For example, in some examples, application information related to the target application may include information about the target application itself. Exemplarily, this may include the target application's running status, the data involved, etc. For instance, if the target application is a telephone application, the application information associated with it may include when the target application was activated, the region of the call being made or received, etc., to facilitate subsequent determination of whether the AI ​​model's functionality is needed for call translation; that is, to facilitate subsequent determination of whether it is an AI scenario based on this application information.

[0067] Furthermore, in some examples, the application information related to the target application may also include information about the application call chain involved in the target application using the AI ​​model. This application call chain may include one or more other applications that the target application traverses before using the AI ​​model. For example, after the Notes app is enabled, if the clipboard app's copy and paste function is also enabled and / or an input method app is launched to input large amounts of text, it may be necessary to enable the AI ​​model's functions to perform text polishing or text translation. Based on this, in this example, the application call chain may include the Notes app as the target application, as well as information including the clipboard app and / or the input method app. For example, it may include the timing and order in which the Notes app, clipboard app, and input method app are enabled, to facilitate subsequent determination of whether the AI ​​model needs to be used to perform functions such as text polishing or text translation. In other words, it facilitates subsequent determination of whether it is an AI scenario based on the information in the application call chain.

[0068] For example, the application call chain may include a user-mode application call chain in the electronic device, specifically including the usage timing information of the user-mode application, or it may include a kernel-mode thread call chain in the electronic device, specifically including the wake-up timing information of the kernel-mode threads. For example, in Figure 3 In the example shown, the application information may specifically include the usage timing information between application A, application B and application C related to the target application, and may also include the usage timing information between kernel-mode threads A, thread B and thread C related to the target application.

[0069] In this embodiment of the application, there can be multiple methods for obtaining the application information, and no limitation is made here. For example, the application information can be obtained based on one or more of the trace tool, perf tool, or logs in the electronic device.

[0070] The trace tool is used to record and analyze events and behaviors of an application during its execution to help understand how the application is running. For example, the specific functions of this trace tool include one or more of the following:

[0071] 1) Record important events during application execution, such as function calls, system calls, and error events;

[0072] 2) Measure and record the time of events to identify latency and performance issues;

[0073] 3) Record and analyze call relationships to show the call relationships between different parts of the application;

[0074] 4) Collect and record data on the application's runtime behavior, such as memory usage and central processing unit (CPU) utilization.

[0075] The perf tool is a performance analysis tool primarily used on Linux systems to analyze the performance of applications and their runtime. For example, the perf tool's specific functions include one or more of the following:

[0076] 1) Collect data from CPU performance counters to help identify performance metrics such as CPU utilization, cache hit rate, and branch hit rate;

[0077] 2) Record the application's running status through periodic sampling, including function calls, execution time, etc., allowing users to observe the application's behavior in different states;

[0078] 3) Supports tracking various system calls, helping to analyze the interaction between applications and the operating system.

[0079] 4) Generate a call graph to visualize function call relationships, facilitating the analysis of application execution flow;

[0080] 5) Generate heatmaps to show which code segments or functions in the application consume more resources, helping developers quickly identify performance bottlenecks.

[0081] Logs are used to record the operating status, events, and activities of electronic devices. Specifically, the log may include one or more log files within the electronic device, such as system logs, application logs, security logs, and transaction logs.

[0082] 3. Sensor data from electronic devices

[0083] In this example, sensor data can be used to describe the environment in which the electronic device is located, such as the external environment of the electronic device.

[0084] The specific content of the sensing data is not limited here. For example, the sensing data may include one or more of the following: image data captured by a camera, depth data detected by a depth sensor, temperature data detected by a temperature sensor, illumination data detected by a light sensor, infrared data detected by an infrared sensor, sound data detected by a microphone or the like, sensing data detected by a laser sensor, and pose information.

[0085] The pose information may include position information and / or attitude information, specifically including one or more of the following: position information detected by a system such as a global positioning system (GPS), acceleration, angular velocity and / or orientation information detected by an inertial measurement unit (IMU) including an accelerometer, a gyroscope and / or a magnetometer.

[0086] Through this sensor data, electronic devices can perceive their external environment and other conditions, which facilitates subsequent scene analysis and other processing.

[0087] As can be seen, in this application embodiment, the application scenario information may include, but is not limited to, one or more of the above information, to describe the current application scenario from one or more aspects, which helps to accurately identify the corresponding scenario and user intent, and provides a richer data foundation for subsequent data processing.

[0088] After obtaining the application scenario information, step 202 can be executed.

[0089] Step 202: Determine whether the electronic device is in an AI scenario based on the application scenario information.

[0090] In this embodiment, before the target application explicitly indicates that the function requiring the AI ​​model needs to be enabled, it is possible to pre-estimate whether the electronic device is in an AI scenario based on application scenario information. For example, the application scenario information may be application scenario information within a time period from a certain historical time point to the current time point, and it can be used to infer whether the application scenario corresponding to the electronic device is an AI scenario.

[0091] In this embodiment, the AI ​​scenario indicates an intention to enable an AI model. In other words, the AI ​​scenario is an estimated scenario that may require the use of an AI model, but it is not actually triggered by the target application to enable the AI ​​model. Instead, it is inferred that the electronic device is highly likely to use an AI model in this AI scenario.

[0092] There are multiple ways to determine whether an application scenario is an AI scenario based on the application scenario information, and no restrictions are imposed here.

[0093] In some embodiments, it can be determined whether an electronic device is in an AI scenario based on application scenario information, preset rules, and / or machine learning models.

[0094] For example, developers can pre-set judgment rules based on experience as preset rules. Then, based on these preset rules and application scenario information, a scenario judgment can be made. If the application scenario information meets the specified conditions in the preset rules, the corresponding application scenario can be determined to be an AI scenario. For instance, a mapping relationship between preset behavioral features and preset sub-results can be constructed based on clustering methods, and this relationship can be converted into conditional branches to obtain preset rules. In this way, the current behavioral features can be extracted from the application scenario information, and based on the conditional branches in the judgment rules, a target judgment result can be determined from multiple preset sub-results. This target judgment result can indicate whether the corresponding application scenario is an AI scenario.

[0095] Alternatively, in some examples, methods such as machine learning models can be used to determine whether the current application scenario is an AI scenario. In this example, the machine learning model can be a lightweight model. The output data of the machine learning model can include multiple candidate judgment results and the probability value corresponding to each candidate judgment result. Each candidate judgment result is used to indicate whether the corresponding application scenario is an AI scenario. Wherein, if the probability value of the candidate judgment result with the highest probability value is higher than a preset threshold, then that candidate judgment result can be used as the target judgment result, which can indicate whether the corresponding application scenario is an AI scenario.

[0096] If the application scenario is determined to be an AI scenario based on the application scenario information, step 203 can be executed.

[0097] Step 203: If it is determined that the electronic device is in an AI scenario based on the application scenario information, the AI ​​model is loaded from the first storage space of the electronic device to the second storage space of the electronic device.

[0098] The first storage space differs from the second storage space, which is used by the target application to enable the AI ​​model's functionality to perform inference tasks.

[0099] In practical scenarios, when the AI ​​model is not enabled on an electronic device, it can be stored in the device's first storage space. For example, this first storage space can be a non-volatile memory such as flash memory or read-only memory (ROM) to reduce the AI ​​model's resource consumption (memory, etc.) and prevent it from interfering with the implementation of other functions in the electronic device.

[0100] When an electronic device needs to enable the functionality of the AI ​​model (specifically, when a target application within the electronic device needs to enable the functionality of the AI ​​model), it typically needs to load the AI ​​model from the first storage space into a second storage space. This allows the AI ​​model to be applied in the second storage space to perform corresponding processing tasks. For example, the second storage space can be the memory of the electronic device, and the specific storage device type can be volatile memory such as DRAM, DDR, or static random-access memory (SRAM). In this way, the process or thread of the target application running in memory can perform corresponding inference tasks using the AI ​​model loaded into memory (i.e., the second storage space).

[0101] In this embodiment, before the target application explicitly indicates that the AI ​​model needs to be enabled (specifically, it may be that a trigger command has been received), the application scenario information can be used to pre-infer whether it is an AI scenario, thereby enabling the pre-loading of the AI ​​model if it is inferred to be an AI scenario.

[0102] In this embodiment of the application, loading the AI ​​model from the first storage space of the electronic device to the second storage space of the electronic device before the target application in the electronic device explicitly instructs to enable the AI ​​model function can specifically mean starting to load the AI ​​model from the first storage space to the second storage space before the target application explicitly instructs to enable the AI ​​model function. At this time, it is not limited to completing the operation of loading the AI ​​model from the first storage space to the second storage space before the target application instructs to enable the AI ​​model function. That is to say, when the target application instructs to enable the AI ​​model function, the operation of loading the AI ​​model from the first storage space to the second storage space can be completed, or the operation of loading the AI ​​model from the first storage space to the second storage space may not have been completed.

[0103] In this embodiment, the time required to load the AI ​​model from the first storage space to the second storage space is determined by the number of parameters of the AI ​​model and the communication resources between the first storage space and the second storage space.

[0104] In some embodiments, the method further includes:

[0105] Within the specified communication resource range, the communication resources between the first storage space and the second storage space are determined based on the resource usage and / or energy consumption requirements of the electronic device;

[0106] Step 203 includes:

[0107] When it is determined that the electronic device is in an AI scenario based on application scenario information, the AI ​​model is loaded from the first storage space to the second storage space through communication resources.

[0108] In this embodiment, the resource usage of the electronic device can reflect information such as the current resource utilization rate and power consumption. Thus, based on the number of AI model parameters and within the communication resource range between the first and second storage spaces, the input / output (I / O) bandwidth meeting preset energy efficiency requirements can be determined as the communication resource between the first and second storage spaces, according to the electronic device's resource usage. This allows the AI ​​model to be loaded with the desired energy efficiency. Therefore, in this embodiment, the bandwidth and other communication resources for loading the AI ​​model can be dynamically controlled, enabling the AI ​​model to be loaded under ideal energy efficiency and / or bandwidth conditions, thereby reducing power consumption during the AI ​​model loading process, saving electronic device resources, and ensuring the operating performance of the electronic device.

[0109] Step 204: Obtain the trigger command.

[0110] Trigger commands are either user-input commands or commands generated by the target application. Trigger commands are used to instruct the target application to enable the AI ​​model's functionality.

[0111] In this embodiment, after triggering the preloading of the AI ​​model, when the target application in the electronic device explicitly instructs the activation of the AI ​​model's function, the AI ​​model preloaded into a second storage space such as memory can be activated to quickly start the AI ​​model's function to perform the corresponding inference task. It is evident that in some embodiments, the time of obtaining the trigger instruction is later than the time when the AI ​​model begins to be loaded from the first storage space to the second storage space. Specifically, it can be determined that the target application explicitly instructs the activation of the AI ​​model's function when the trigger instruction is obtained.

[0112] The trigger command can be user-inputted, for example, by clicking on a trigger item for the AI ​​function through the target application's interface. Alternatively, the target application can automatically generate the trigger command based on its own functionality. This trigger command instructs the target application to enable the AI ​​model's functionality to perform inference tasks. Specifically, it can instruct the launch of a specified process or thread, which, when running, needs to invoke the AI ​​model and perform corresponding inference tasks. Launching the specified process or thread can be done by waking up an existing process, that is, restoring a process or thread that is in a standby, suspended, or low-power state to an active state; or it can be done by creating and executing a completely new process or thread. The specified process or thread can run in a secondary storage space.

[0113] Step 205: Based on the trigger command, run the AI ​​model that has been loaded into the second storage space to enable the AI ​​model's function to perform inference tasks.

[0114] For example, in Figure 3 In the example shown, when the application scenario is determined to be an AI scenario based on the application scenario information, during the preloading of the AI ​​model, the target application can wake up the specified thread based on the trigger command. After the thread is woken up, the AI ​​model that has been loaded into the second storage space DDR can be quickly run through the thread to perform the corresponding inference task, without having to perform the AI ​​model preloading operation after waking up the thread. This effectively shortens the processing time and reduces latency performance indicators such as TTFT.

[0115] As can be seen, in this embodiment, scene perception can be achieved more accurately based on one or more aspects of application scenario information, such as interaction information between electronic devices and other devices, application information related to the target application, and sensor data of the electronic device. This allows for the estimation of user intent and assessment of whether the scenario indicates an intention to enable the AI ​​model. Therefore, if an AI scenario is identified, the AI ​​model can be pre-loaded before the target application in the electronic device explicitly instructs to enable the AI ​​model based on a trigger command. This pre-loading approach reduces the loading time of the AI ​​model after the target application instructs to enable it, effectively shortening the corresponding inference time and significantly improving latency performance such as TTFT. Furthermore, this method avoids complex hardware and software scheduling logic and the model accuracy reduction issues caused by model compression techniques such as static and dynamic pruning. The scheduling logic is simple and highly executable.

[0116] The effects of any of the above embodiments of this application will be illustrated below through a test of an exemplary scenario.

[0117] In traditional AI model usage scenarios, the latency experienced between the electronic device instructing the activation of the AI ​​model and the output of the first token can be comprised of three parts, specifically:

[0118] The latency of loading the AI ​​model from non-volatile memory into memory, the latency of prefilling the AI ​​model, and the latency of inference by the AI ​​model on the first token.

[0119] AI model prefill refers to the process of using an artificial intelligence model to intelligently predict and automatically fill in certain information for certain fields or data items entered by the user, based on context or the user's historical behavior. For example, this information can be text, options, form fields, etc., which means that the model can understand the context, infer the user's needs, and generate reasonable fill content.

[0120] Taking a terminal device currently deployed with an AI model as an example, an exemplary case of the latency experienced by the terminal device from instructing the AI ​​model to be enabled to the output of the first token is as follows:

[0121] 1) Latency of loading AI model from non-volatile memory Flash to memory DDR: With a Flash bandwidth of 5GBps, the latency of loading a 1.5B-INT4 quantized model (0.9B) from non-volatile memory Flash to memory DDR is approximately 180ms.

[0122] 2) AI model prefill latency: Under the conditions of 51.2GBps DDR bandwidth and 18T computing power, the prefill time of a 1.5B AI model under the computing bound of 50 to 1K tokens is approximately 15 to 160ms.

[0123] 3) Inference latency of the AI ​​model for the first token: Under the conditions of 51.2GBps DDR bandwidth and 18T computing power, the latency of the 1.5B AI model for inference and output of the first token is 15-20ms.

[0124] Furthermore, depending on the size of the AI ​​model and / or the computing power, the latency of the AI ​​model preloading scheme using any of the above embodiments of this application is compared to that of the traditional scheme, such as... Figures 4a to 4b As shown. Among them, through Figure 4a As the example shown illustrates, with a fixed computing power of 18T, as the size of the AI ​​model increases, the latency of the AI ​​model preloading scheme is significantly reduced compared to the traditional scheme, specifically, by 46.3% to 52.6%. Furthermore, through... Figure 4bAs shown in the example, with a fixed AI model size of 1.5B, the latency of the AI ​​model preloading scheme is significantly reduced compared to the traditional scheme as computing power increases. Specifically, it can be reduced by 34.8% to 72%.

[0125] As can be seen, the AI ​​model preloading scheme of any of the above embodiments of this application can reduce or even eliminate the latency of loading the AI ​​model from non-volatile memory Flash to memory DDR (for example, it may bring a benefit of 50% to 85.7%), and improve the overall processing efficiency.

[0126] The above describes the reasoning task execution method provided by the embodiments of this application from multiple aspects. The following describes the reasoning task execution device provided by the embodiments of this application in conjunction with the accompanying drawings.

[0127] like Figure 5 As shown, this application provides an inference task execution device 50, which can be applied to electronic devices. The inference task execution device 50 includes:

[0128] The acquisition module 501 is used to: acquire application scenario information about the electronic device;

[0129] The processing module 502 is used to: when it is determined that the electronic device is in an artificial intelligence (AI) scenario based on application scenario information, load the AI ​​model from the first storage space of the electronic device to the second storage space of the electronic device. The AI ​​scenario indicates that there is an intention to enable the AI ​​model. The second storage space is used for the target application to enable the AI ​​model to perform inference tasks. The first storage space is different from the second storage space.

[0130] The acquisition module 501 is used to: acquire trigger instructions, which are instructions input by the user or instructions generated by the target application, and the trigger instructions are used to instruct the target application to enable the function of the AI ​​model;

[0131] The processing module 502 is used to: run the AI ​​model that has been loaded into the second storage space based on the trigger command, so as to enable the AI ​​model to perform inference tasks.

[0132] Optionally, the time to obtain the trigger instruction is later than the time to start loading the AI ​​model from the first storage space to the second storage space.

[0133] Optionally, the first storage space is located in the non-volatile memory of the electronic device, and the second storage space is located in the memory of the electronic device.

[0134] Optionally, the application scenario information includes one or more of the following: interaction information between the electronic device and other devices, application information related to the target application, and sensor data of the electronic device.

[0135] Optionally, the processing module 502 is used for:

[0136] Within the specified communication resource range, the communication resources between the first storage space and the second storage space are determined based on the resource usage and / or energy consumption requirements of the electronic device;

[0137] When it is determined that the electronic device is in an AI scenario based on application scenario information, the AI ​​model is loaded from the first storage space to the second storage space through communication resources.

[0138] Optionally, the processing module 502 is used for:

[0139] Based on application scenario information, and using preset rules and / or machine learning models, determine whether the electronic device is in an AI scenario.

[0140] Figure 6 The diagram shown is a possible logical structure schematic of an electronic device 60 provided in an embodiment of this application. This electronic device 60 is used to implement the functions of the electronic devices involved in any of the above embodiments. The electronic device 60 includes: a memory 601, a processor 602, a communication interface 603, and a bus 604. The memory 601, processor 602, and communication interface 603 are interconnected via the bus 604.

[0141] The memory 601 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 601 may store a program, and when the program stored in the memory 601 is executed by the processor 602, the processor 602 and the communication interface 603 are used to perform one or more steps in the above-described inference task execution method embodiment.

[0142] The processor 602 can be a central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), digital signal processing (DSP), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any combination thereof, to execute relevant programs to achieve the functions required by the acquisition module and processing module in the inference task execution device of the above embodiments, or to execute one or more steps in the method embodiments of this application. The steps of the method disclosed in the embodiments of this application can be executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 601. The processor 602 reads the information in memory 601 and, in conjunction with its hardware, executes one or more steps in the above-described inference task execution method embodiments.

[0143] The communication interface 603 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the electronic device 60 and other devices or communication networks.

[0144] Bus 604 enables the transmission of information between various components of electronic device 60 (e.g., memory 601, processor 602, and communication interface 603). Bus 604 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0145] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device performs the aforementioned... Figure 6 The steps performed by the processor in the process.

[0146] In another embodiment of this application, a computer program product is also provided, which includes computer-executable instructions stored in a computer-readable storage medium; when the processor of the device executes the computer-executable instructions, the device performs the above-described... Figure 6 The steps performed by the processor in the process.

[0147] In another embodiment of this application, a chip system is also provided, the chip system including a processor for implementing the above. Figure 6 The steps performed by the processor. In one possible design, the chip system may also include a memory for storing program instructions and data necessary for data writing. The chip system may consist of chips or may include chips and other discrete components.

[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for performing a reasoning task, characterized in that, Applied to electronic devices, the method includes: Obtain application scenario information about the electronic device; When it is determined that the electronic device is in an artificial intelligence (AI) scenario based on the application scenario information, the AI ​​model is loaded from the first storage space of the electronic device to the second storage space of the electronic device. The AI ​​scenario indicates that there is an intention to enable the AI ​​model. The second storage space is used by the target application to enable the function of the AI ​​model to perform inference tasks. The first storage space is different from the second storage space. Obtain a trigger instruction, which is either an instruction input by the user or an instruction generated by the target application, and the trigger instruction is used to instruct the target application to enable the function of the AI ​​model; Based on the trigger command, the AI ​​model that has been loaded into the second storage space is run to enable the AI ​​model's functionality to perform the inference task.

2. The method according to claim 1, characterized in that, The time of obtaining the trigger instruction is later than the time when the AI ​​model is started being loaded from the first storage space to the second storage space.

3. The method according to claim 1 or 2, characterized in that, The first storage space is located in the non-volatile memory of the electronic device, and the second storage space is located in the memory of the electronic device.

4. The method according to any one of claims 1-3, characterized in that, The application scenario information includes one or more of the following: interaction information between the electronic device and other devices, application information related to the target application, and sensor data of the electronic device.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Within the specified communication resource range, the communication resources between the first storage space and the second storage space are determined based on the resource usage and / or energy consumption requirements of the electronic device; When the application scenario information determines that the electronic device is in an artificial intelligence (AI) scenario, loading the AI ​​model from the first storage space of the electronic device to the second storage space of the electronic device includes: When it is determined that the electronic device is in the AI ​​scenario based on the application scenario information, the AI ​​model is loaded from the first storage space to the second storage space through the communication resources.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Based on the application scenario information, and using preset rules and / or machine learning models, it is determined whether the electronic device is in the AI ​​scenario.

7. A reasoning task execution device, characterized in that, Applied to electronic devices, the device includes an acquisition module and a processing module: The acquisition module is used to: acquire application scenario information about the electronic device; The processing module is used to: when it is determined that the electronic device is in an artificial intelligence (AI) scenario based on the application scenario information, load the AI ​​model from the first storage space of the electronic device to the second storage space of the electronic device, wherein the AI ​​scenario indicates an intention to enable the AI ​​model, and the second storage space is used for the target application to enable the function of the AI ​​model to perform inference tasks, and the first storage space is different from the second storage space; The acquisition module is used to: acquire a trigger instruction, wherein the trigger instruction is an instruction input by the user or an instruction generated by the target application, and the trigger instruction is used to instruct the target application to enable the function of the AI ​​model; The processing module is used to: run the AI ​​model that has been loaded into the second storage space based on the trigger instruction, so as to enable the function of the AI ​​model to perform the inference task.

8. The apparatus according to claim 7, characterized in that, The time of obtaining the trigger instruction is later than the time when the AI ​​model is started being loaded from the first storage space to the second storage space.

9. The apparatus according to claim 7 or 8, characterized in that, The first storage space is located in the non-volatile memory of the electronic device, and the second storage space is located in the memory of the electronic device.

10. The apparatus according to any one of claims 7-9, characterized in that, The application scenario information includes one or more of the following: interaction information between the electronic device and other devices, application information related to the target application, and sensor data of the electronic device.

11. The apparatus according to any one of claims 7-10, characterized in that, The processing module is used for: Within the specified communication resource range, the communication resources between the first storage space and the second storage space are determined based on the resource usage and / or energy consumption requirements of the electronic device; When it is determined that the electronic device is in the AI ​​scenario based on the application scenario information, the AI ​​model is loaded from the first storage space to the second storage space through the communication resources.

12. The apparatus according to any one of claims 7-11, characterized in that, The processing module is used for: Based on the application scenario information, and using preset rules and / or machine learning models, it is determined whether the electronic device is in the AI ​​scenario.

13. An electronic device, characterized in that, The electronic device includes at least one processor, a memory, and instructions stored in the memory and executable by the at least one processor, wherein the at least one processor executes the instructions to implement the steps of the method according to any one of claims 1-6.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-6.

15. A computer program product containing instructions, characterized in that, When the instructions are executed by the processor, the method described in any one of claims 1-6 is implemented.

Images