Equipment model deployment method and device, end side equipment and computer readable storage medium

By optimizing the hardware characteristics and application scenarios of smart home devices, compressing large AI models and dynamically assigning processing tasks, the hardware resource limitation problem of large AI models for smart home devices is solved, and efficient model deployment and operation performance is achieved.

CN119067187APending Publication Date: 2024-12-03QINGDAO HAIER TECH +3

Patent Information

Application Number
CN202411014066.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Due to limited hardware resources, smart home devices cannot directly deploy and run large AI models learned from transfers, and the existing technology has not specifically optimized for large models deployed on terminal devices, resulting in the quality of push flow cannot meet actual needs.

Method used

By compressing the deployment model based on the hardware characteristics and application scenarios of the end-side device, a lightweight target model is obtained, and processing tasks are dynamically allocated based on the model inference framework to ensure full utilization of hardware resources and performance improvement.

Benefits of technology

While reducing model complexity and operating power consumption, we maximize model performance, make full use of end-side equipment hardware resources, and improve performance and computing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067187A_ABST
    Figure CN119067187A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computers, and discloses an equipment model deployment method, which comprises the following steps of: performing compression processing on a to-be-deployed model according to hardware characteristics and / or an application scene of end-side equipment to obtain a lightweight target model; dynamically allocating processing tasks of the lightweight target model based on a model reasoning framework; wherein the model reasoning framework and the lightweight target model are deployed on end side equipment. According to the method, the complexity and operation power consumption of the target model are reduced, and the performance of the model is kept to the maximum extent; therefore, full utilization of hardware resources of the end-side equipment is ensured, and performance improvement is facilitated. The invention further discloses an equipment model deployment device, the end side equipment and a computer readable storage medium.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and for example, to a device model deployment method and apparatus, a terminal device, and a computer-readable storage medium. Background Art

[0002] With the development of deep learning technology and the popularization of the Internet of Things, various smart home terminal devices have begun to gradually use AI technology to improve user experience. Smart home devices are limited by hardware resources such as CPU (Central Processing Unit) / GPU (Graphics Processing Unit) computing power, memory capacity and power consumption, and cannot directly deploy and run large AI (Artificial Intelligence Plus) models transferred from local learning. How to achieve efficient deployment and operation of large models with limited hardware resources is a problem that needs to be solved at present.

[0003] The TensorRT framework is used in related technologies for model optimization and deployment. The TensorRT framework supports both DNN (Deep-Learning Neural Network) and CNN (Convolutional Neural Network) models, and provides floating-point and integer precision levels for calculations. The framework works by parsing the original trained AI model, reducing or even removing duplicate or unnecessary elements in the network structure, and saving the network weights in half-precision or INT8 format to reduce size. The optimized framework can package these pre-processed elements together into a new session for streaming.

[0004] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:

[0005] The relevant technology has not been specifically optimized for large models deployed and running on terminal devices. Due to factors such as network structure and hardware performance, there is a problem that the streaming quality cannot meet actual needs.

[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0008] The embodiments of the present disclosure provide a device model deployment method and apparatus, a terminal device, and a computer-readable storage medium to implement specific optimization of large models on the terminal device.

[0009] In some embodiments, the method includes: compressing the model to be deployed according to the hardware characteristics and / or application scenarios of the end-side device to obtain a lightweight target model; dynamically allocating processing tasks of the lightweight target model based on the model inference framework; wherein the model inference framework and the lightweight target model are deployed on the end-side device.

[0010] In some embodiments, the apparatus includes: a processor and a memory storing program instructions, and the processor is configured to execute the aforementioned model deployment method for end-side devices when running the program instructions.

[0011] In some embodiments, the terminal side device includes: a terminal side device body; and a model deployment device for the terminal side device as described above, which is installed on the terminal side device body.

[0012] In some embodiments, the computer-readable storage medium stores program instructions, and when the program instructions are executed, the computer executes the model deployment method for the terminal device as described above.

[0013] The device model deployment method and apparatus, terminal device, and computer-readable storage medium provided in the embodiments of the present disclosure can achieve the following technical effects:

[0014] The disclosed embodiment comprehensively considers the hardware characteristics and / or application scenario requirements of the end-side device, and compresses the model to be deployed to obtain a lightweight target model. In this way, the performance of the model is maintained to the maximum extent while reducing the complexity and operating power consumption of the model to be deployed. Based on the model reasoning framework, the processing tasks of the model are dynamically allocated; thereby ensuring the full utilization of the hardware resources of the end-side device and helping to improve performance.

[0015] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein:

[0017] Figure 1 is a schematic diagram of a device model deployment method provided by an embodiment of the present disclosure;

[0018] Figure 2 is a schematic diagram of another device model deployment method provided by an embodiment of the present disclosure;

[0019] Figure 3 is a schematic diagram of another device model deployment method provided by an embodiment of the present disclosure;

[0020] Figure 4 is a schematic diagram of another device model deployment method provided by an embodiment of the present disclosure;

[0021] Figure 5 is a schematic diagram of a device model deployment apparatus provided by an embodiment of the present disclosure;

[0022] Figure 6 It is a schematic diagram of a terminal-side device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0024] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0025] Unless otherwise stated, the term "plurality" means two or more.

[0026] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0027] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0028] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0029] In the embodiments of the present disclosure, the end-side device includes a smart home appliance and / or a terminal device connected to the smart home appliance. Among them, the smart home appliance refers to a home appliance product formed by introducing a microprocessor, sensor technology, and network communication technology into home appliances, which has the characteristics of intelligent control, intelligent perception, and intelligent application. The operation process of the smart home appliance often relies on the application and processing of modern technologies such as the Internet of Things, the Internet, and electronic chips. For example, the smart home appliance can be connected to an electronic device to enable the user to remotely control and manage the smart home appliance. The terminal device refers to an electronic device with a wireless connection function. The terminal device can communicate and connect with the above smart home appliances by connecting to the Internet, or it can directly communicate and connect with the above smart home appliances through Bluetooth, Wi-Fi, etc. In some embodiments, the terminal device is, for example, a mobile device, a computer, etc., or any combination thereof. Mobile devices may include, for example, mobile phones, smart home devices, wearable devices, smart mobile devices, virtual reality devices, etc., or any combination thereof.

[0030] Combination Figure 1 As shown, the embodiment of the present disclosure provides a device model deployment method, including:

[0031] S101, the terminal device compresses the model to be deployed according to the hardware characteristics and / or application scenarios of the device to obtain a lightweight target model.

[0032] S102, the terminal side device dynamically allocates processing tasks of the lightweight target model based on the model reasoning framework; wherein the model reasoning framework and the lightweight target model are deployed on the terminal side device.

[0033] Here, the hardware characteristics of the end-side device include characteristics such as processor, memory, storage and network. The application scenarios of the end-side device include but are not limited to speech / text recognition, image recognition and video stream processing. The target model includes large models such as deep learning models and neural network models. The compression processing of the target model can be performed through an automated tool chain, which can adjust the network structure of the model and compress the model. Among them, the automated tool chain refers to the integration of various model processing tools to form a tool chain, and each model processing tool can be determined based on the hardware characteristics and application scenarios of the end-side device. Specifically, the model to be deployed can be compressed based on one or two of the hardware characteristics of the end-side device and the application scenarios. As an example, due to hardware limitations, 4-bit quantization technology can be used, such as LPDNN (Low-power DNN) or TinyML and other tools; and / or, low-level convolution tools are used to replace high-complexity convolutions in the target model with equivalent low-complexity convolutions; and / or, weight pruning, network pruning and other tools are used. As another example, when the application scenario is image recognition or video frame recognition processing, in order to ensure the accuracy of the processing, it is necessary to retain the network structure with strong rotation invariance, and the network structure with strong rotation invariance cannot be adjusted to a network structure with weak rotation invariance. In this way, when obtaining a lightweight target model, the hardware characteristics of the terminal device and / or the application scenario requirements are comprehensively considered, while reducing the complexity and operating power consumption of the target model, the performance of the model is maintained to the maximum extent.

[0034] Furthermore, based on the model reasoning framework of the device layout on the end side, the processing tasks of the lightweight target model are dynamically allocated to the processor of the device on the end side. Among them, the obtained lightweight target model is also deployed on the device on the end side. As an example, the model reasoning framework can be a PowerInfer framework, or a TensorRT framework, etc. Specifically, the processor of the device on the end side includes a GPU and a CPU; the processing tasks of the two are different. Based on the model reasoning framework, the tasks of the lightweight target model are allocated, the lightweight target model is analyzed, and the sparse part and the dense part are identified. Further, based on the resource usage of the processor, the data preprocessing and lightweight computing tasks corresponding to the sparse part are dynamically allocated to the CPU, and the large-scale computing tasks corresponding to the dense part are allocated to the GPU. Among them, dynamically allocating processing tasks refers to dynamically adjusting the allocation of processing tasks according to the resource utilization of the hardware (such as the resource utilization of the processor). As an example, when the GPU resources are insufficient, the tasks allocated to the GPU can be dynamically adjusted, that is, some computing-intensive tasks are temporarily allocated to the CPU; when the GPU resources are sufficient, the previous allocation method is restored. In this way, the reasonable allocation of lightweight model tasks is ensured so that the hardware resources of the terminal device can be fully utilized.

[0035] The model deployment method for end-side devices provided by the embodiment of the present disclosure can comprehensively consider the hardware characteristics and application scenario requirements of the end-side devices, and compress the target model to obtain a lightweight target model. In this way, while reducing the complexity and operating power consumption of the target model, the performance of the model is maintained to the maximum extent. And based on the model reasoning framework (the PowerInfer framework is used in the example of this disclosure), the processing tasks of the model are dynamically allocated; thereby ensuring the full utilization of the hardware resources of the end-side device and helping to improve performance.

[0036] Optionally, in step S102, the terminal device dynamically allocates processing tasks of the lightweight target model based on the model reasoning framework, including:

[0037] Based on the PowerInfer framework, the end-side device classifies the processing tasks of the lightweight target model and assigns the processing tasks to the corresponding processors based on the classification.

[0038] The terminal device dynamically adjusts the initial task weight of the processor according to the perceived resource utilization of each processor to adjust the distribution of processing tasks.

[0039] Here, the model structure (including each layer and node, etc.), parameters and execution process of the lightweight target model are analyzed based on the PowerInfer framework to identify the sparse and dense parts of the model. Based on the sparse and dense parts, the processing tasks are divided into two categories: one is the processing tasks for data preprocessing and lightweight calculations (this type of task is generally included in the sparse part of the model), and the other is the processing tasks for large-scale complex calculations (this type of task is generally included in the dense part of the model). Then the tasks corresponding to the dense part are assigned to the GPU, and the tasks corresponding to the sparse part are assigned to the CPU.

[0040] After the processing tasks are assigned, each processor processes the assigned tasks. The resource utilization of each processor is perceived in real time, where the resource utilization of the processor includes the computing power, memory occupancy, power consumption, and occupancy of the processor. Then, according to the perceived resource utilization of each processor, the task weight of the processor is dynamically adjusted to adjust the allocation of processing tasks. As an example, if the resource utilization of the GPU indicates that the GPU resources are sufficient, the proportion of GPU processing tasks can be increased. If the resource utilization of the GPU indicates that the GPU resources are appropriate, the current proportion of GPU processing tasks is maintained. In this way, by real-time perception and dynamic adjustment of the allocation of processing tasks, the performance of the terminal-side device is continuously improved, making it more robust and efficient.

[0041] Optionally, the terminal device dynamically adjusts the task weight of the processor according to the perceived resource utilization of each processor, including:

[0042] The terminal device obtains the initial task weight of each processor, wherein the initial task weight is determined based on the static characteristics of the terminal device.

[0043] When the resource utilization of the processor indicates that the occupancy rate of the processor is less than the first threshold, the terminal device upwardly corrects the initial task weight corresponding to the processor, and restores the initial task weight when the latest occupancy rate is greater than the second threshold.

[0044] Here, the static characteristics of the end-side device include hardware specifications, operating system, power supply, etc. Appropriate initial task weights are assigned to each processor based on the static characteristics. In addition, the threshold value ranges corresponding to each processor can be different or the same. Generally, the values ​​of the first threshold and the second threshold of the GPU are smaller than those of the CPU. As an example, the value range of the first threshold corresponding to the CPG is 30% to 40%, and the value range of the second threshold is 50% to 60%. The value range of the first threshold corresponding to the GPU is 25% to 35%, and the value range of the second threshold is 45% to 55%. In this way, the processing task allocation is dynamically adjusted based on the occupancy rate of each processor so that the processor always maintains better performance and computing efficiency.

[0045] Combination Figure 2 As shown, the embodiment of the present disclosure provides another device model deployment method, including:

[0046] S101, the terminal device compresses the model to be deployed according to the hardware characteristics and / or application scenarios of the device to obtain a lightweight target model.

[0047] S102, the terminal side device dynamically allocates processing tasks of the lightweight target model based on the model reasoning framework; wherein the model reasoning framework and the lightweight target model are deployed on the terminal side device.

[0048] S203: When the CPU sends data to the GPU, the terminal device obtains the interconnection pressure of the data bus device.

[0049] S204, the terminal side device adjusts the bandwidth of sending data according to the interconnection pressure; and adjusts the current task weight of each processor.

[0050] Among them, the terminal side devices include a GPU and a CPU connected through a PCIe device.

[0051] Here, the GPU and the CPU are connected through a data bus device (such as a PCIe device) so that the CPU can transmit data to the GPU. The interconnection pressure of the PCIe device refers to the data transmission load on the PCIe bus. A lower data transmission load indicates that the data transmission is sparser and the interconnection pressure is smaller; conversely, a higher data transmission load indicates that the data transmission is denser and the interconnection pressure is greater. The level of the transmission load can be determined by indicators such as the bus utilization rate and bandwidth utilization rate. Therefore, the utilization rate, bandwidth utilization rate and other indicators of the PCIe bus are detected in real time to obtain the interconnection pressure of the PCIe device. Among them, the higher the utilization rate of the PCIe bus and the higher the bandwidth utilization rate, the higher the interconnection pressure.

[0052] Further, according to the interconnection pressure, the bandwidth of the sent data is adjusted. Wherein adjusting the bandwidth of the sent data means determining the target bandwidth based on the interconnection pressure before each data transmission. After adjusting the bandwidth to the target bandwidth, the data is sent. Generally, the smaller the interconnection pressure, the higher the level of the target bandwidth, that is, a higher bandwidth is used for data transmission. As an example, if the current bandwidth is 1Gbps and the interconnection pressure is small, the target bandwidth can be adjusted to 10Gbps. In addition, according to the interconnection pressure, the current task weight of each processor is adjusted, including: when the interconnection pressure is large, the current task weight of each processor is reduced. When the interconnection pressure is moderate or small, the current task weight of each processor is maintained. Wherein, the magnitude of the interconnection pressure can be characterized by a preset first pressure, a second pressure, etc. As an example, the pressure magnitude is characterized by bandwidth utilization (the ratio of actual usage within a preset time to the theoretical maximum bandwidth), and the first pressure can be a percentage such as 40%. In this way, the task weight or the bandwidth of the sent data is adjusted through interconnection pressure monitoring. Thereby, the optimal computing efficiency of the end-side device can be maintained, while preventing damage or performance degradation of the PCIe device caused by excessive pressure.

[0053] Optionally, the end-side device adjusts the current task weight of each processor including: when the interconnection pressure is greater than the second pressure, the end-side device preferentially adjusts the task weight of the GPU, or the end-side device adjusts the current task weight of each processor based on the attributes of the processing task and the operating status of the processor.

[0054] Here, the second pressure is a critical value used to characterize that the interconnection pressure is large. When the interconnection pressure is greater than the second pressure, the resources of the end-side device processor may be insufficient. Therefore, the task weights of each processor can be appropriately adjusted. The task weight of the GPU can be adjusted preferentially to alleviate the interconnection pressure of the PCIe device. Alternatively, the current task weights of each processor can be further adjusted in combination with the running state of the processor and the attributes of the processing tasks to be assigned. The running state of the processor includes information such as computing pressure, memory utilization, and battery; the attributes of the processing tasks to be assigned include computing amount, priority, etc. Combined with these two pieces of information, the task weight is adjusted. As an example, the running state of the processor indicates that the resources of the processor are insufficient, and / or the attributes of the processing tasks to be assigned indicate that the computing amount is large, then the task weight needs to be adjusted to avoid the processor from taking on too many computing tasks and data transmission work. At the same time, the adjustment range of the current task weight can also be determined based on the running state of the processor and the attributes of the processing tasks to be assigned. The less the resources of the processor, and / or the greater the computing amount of the processing tasks to be assigned, the greater the corresponding adjustment range.

[0055] Optionally, the lower the transmission delay of the PCIe device, the greater the interconnection pressure. Therefore, when adjusting the current task weight of each processor, the weight adjustment can be made specifically for the processing tasks that require a large amount of data transmission (i.e., reducing the weight of such processing tasks), or, based on the task attributes (such as computing volume, priority, expected completion time, etc.), the processing tasks with lower task attributes (such as large computing volume, low priority, and long expected completion time) can be delayed.

[0056] Optionally, in step S204, the end-side device adjusts the bandwidth of sending data according to the interconnection pressure, including:

[0057] When the interconnection pressure is less than the first pressure, the end-side device adjusts the bandwidth of the data to be sent to a high-level bandwidth.

[0058] When the interconnection pressure is greater than the second pressure, the end-side device adjusts the bandwidth of the data to be sent to a lower bandwidth. The first pressure is less than the second pressure.

[0059] As mentioned above, the values ​​of the first pressure and the second pressure are percentage values. As an example, the value range of the first pressure is 30% to 40%, and the value range of the second pressure is 55% to 65%. The bandwidth can determine the maximum bandwidth based on the characteristics of the end-side device; then the bandwidth is divided into multiple levels based on the data transmission requirements, and each level corresponds to a bandwidth range. Then, before the data is sent, it is determined whether to adjust the bandwidth in combination with the interconnection pressure. As an example, the bandwidth is divided into the first bandwidth, the second bandwidth and the third bandwidth; the level of the first bandwidth is higher than the second bandwidth, and the level of the second bandwidth is higher than the third bandwidth. Assuming that the bandwidth of the current data transmission is the second bandwidth, and at the same time, it is detected that the interconnection pressure of the PCIe device is less than the first pressure, the bandwidth of the data to be sent is adjusted to the bandwidth of the previous higher level, that is, the first bandwidth. In this way, when the interconnection pressure is small, a higher level of bandwidth is used for data transmission to increase the transmission rate. When the interconnection pressure is large, a lower level of bandwidth is used for data transmission to avoid overloading the PCIe device.

[0060] Optionally, in step S203, when the CPU sends data to the GPU, after obtaining the interconnection pressure of the PCIe device, the terminal device further includes:

[0061] When the interconnection pressure exceeds the limit pressure, the end-side device stops allocating processing tasks.

[0062] Here, the limit pressure refers to the maximum interconnect pressure allowed by the PCIe device; when the interconnect pressure is greater than or equal to the limit pressure, the PCIe device is at risk of damage. In order to avoid damage, the allocation of processing tasks is temporarily stopped at this time. After the interconnect pressure returns to the limit pressure, the processing tasks are allocated with a task weight less than before the stop, or the previous task weight is maintained but the allocation of non-urgent processing tasks is suspended. In this way, even if the interconnect pressure is restored, the task allocation is appropriately reduced to fully relieve the interconnect pressure. This avoids damage to the PCIe device or performance degradation.

[0063] Optionally, in step S101, the terminal device compresses the target model to be deployed according to the hardware characteristics and application scenarios of the device, including:

[0064] The end-side device matches the model compression processing solution in the preset database according to the hardware characteristics and application scenarios of the device.

[0065] The end-side device compresses the target model to be deployed according to the matching compression processing solution.

[0066] Here, the preset database is based on the big data of the historical deployment model, wherein the big data of the historical deployment model includes the end-side device type and hardware characteristics, application scenarios, and deployment information of the deep learning model (such as model processing, training, and other information). When the target model is deployed on the end-side device in the embodiment of the present disclosure, the hardware characteristics and application scenarios of the device are input into the preset database, and the deployment information of the deep learning model with the highest index match. The matched deployment information is used as the compression processing scheme for the target model; the target model is compressed. Among them, compressing the target model refers to using the tool chain in the matched deployment information to prune the target model and the inverse network, quantize, and use advanced compilation optimization techniques.

[0067] After the target model is compressed, the lightweight target model can be optimized by using the change rules between hardware characteristics and load. The change rules between hardware characteristics and load refer to collecting and analyzing data related to the end-side device, and training and optimizing the lightweight target model after data processing, so that the model converges more stably and achieves optimal performance.

[0068] Combination Figure 3 As shown, the embodiment of the present disclosure provides another device model deployment method, including:

[0069] S101, the terminal device compresses the model to be deployed according to the hardware characteristics and / or application scenarios of the device to obtain a lightweight target model.

[0070] S102, the terminal side device dynamically allocates processing tasks of the lightweight target model based on the model reasoning framework; wherein the model reasoning framework and the lightweight target model are deployed on the terminal side device.

[0071] S305, the terminal device calls the hybrid reasoning engine built under the model reasoning framework.

[0072] S306: The terminal side device uses the hybrid inference engine to obtain the memory usage efficiency of the terminal side device, and optimizes the allocation of processing tasks based on the memory usage efficiency.

[0073] Here, the hybrid inference engine built under the PowerInfer framework refers to a hybrid inference engine built using neuron-aware sparse operators and used in the local memory of the end-side device. Neuron-aware sparse operators can identify sparse data in lightweight target models and interact with neurons with sparse properties. This allows flexible management of calculations in local memory. Therefore, the memory usage efficiency of the end-side device can be determined using the hybrid inference engine; and the allocation of processing tasks can be optimized based on the memory usage efficiency. Among them, the memory of the end-side device refers to the local memory of the buffer of each processor of the end-side device. When the memory usage efficiency of the processor is high, the allocation of processing tasks can be reduced to reduce computing pressure. As an example, when the GPU buffer resources are seriously insufficient, some computing tasks are transferred to the CPU for processing, so that the GPU buffer can retain more storage space load and help reduce computing complexity.

[0074] Optionally, in step S303, a hybrid reasoning engine under the model reasoning framework is constructed in the following manner:

[0075] The end-side device designs a neuron-aware sparse operator based on the network structure of the lightweight target model, input data, and hardware characteristics of the end-side device.

[0076] The end-side device builds and optimizes a hybrid inference engine that supports sparse operators based on the hardware capabilities of the device and the network requirements of the lightweight target model.

[0077] Here, the sparse operator of neuron perception is selected comprehensively based on the network structure of the comprehensive model, the input data and the hardware characteristics of the end-side device. Due to the hardware limitations of the end-side device, especially the memory limitations, the sparse operator needs to be lightweight. At the same time, the sparse operator will affect the network structure and parameter quantity of the model. For example, the applicability of sparse operators varies depending on the nature of the data. Data of certain properties are more susceptible to sparsification, so the nature of the input data needs to be considered when designing sparse operators. As an example, if the data nature is an image, then a suitable sparse operator is selected based on the sparse characteristics of the image. Similarly, some network structures are sensitive to sparse operators, while others are not suitable for sparse operators. Therefore, the influence of the network structure must also be considered when designing sparse operators. As an example, the network structure includes parameters such as the number of layers, node connection method and parameter scale. The sparse operator can be selected in combination with the specific parameters of the network structure. Combining the above three factors, a sparse operator is selected.

[0078] Then, based on the hardware capabilities of the end-side device and the network requirements of the lightweight target model, a hybrid inference engine that supports sparse operators is built. The hardware capabilities of the end-side device include the computing power of the device (such as the computing performance of the GPU and CPU), memory size, storage speed, parallel computing capabilities, and supported instruction sets. Network requirements refer to the depth, width, number of layers, number of neurons, connection mode, activation function, loss function, etc. of the lightweight target model. After fully determining the hardware capabilities of the end-side device and the network requirements of the lightweight target model, the sparse operator is integrated into the PowerInfer framework to form a hybrid inference engine. Finally, after the GPU-CPU hybrid inference engine is built, the inference speed, power consumption, and resource utilization are evaluated. Based on the evaluation results, the parameters and design of the engine are adjusted and optimized so that the engine's allocation of processor tasks meets the performance requirements of the end-side device and achieves an efficient inference process.

[0079] Combination Figure 4 As shown, the embodiment of the present disclosure provides another device model deployment method, including:

[0080] The present disclosure provides another model deployment method for a terminal device, including:

[0081] S101: The terminal device compresses the model to be deployed according to the hardware characteristics and application scenarios of the device to obtain a lightweight target model.

[0082] S102, the terminal side device dynamically allocates processing tasks of the lightweight target model based on the model reasoning framework; wherein the model reasoning framework and the lightweight target model are deployed on the terminal side device.

[0083] S203: When the CPU sends data to the GPU, the terminal device obtains the interconnection pressure of the PCIe device.

[0084] S204, the terminal side device adjusts the bandwidth of sending data according to the interconnection pressure; and adjusts the current task weight of each processor.

[0085] Among them, the terminal side devices include a GPU and a CPU connected through a PCIe device.

[0086] S305, the terminal device calls the hybrid reasoning engine built under the model reasoning framework.

[0087] S306: The terminal side device uses the hybrid inference engine to obtain the memory usage efficiency of the terminal side device, and optimizes the allocation of processing tasks based on the memory usage efficiency.

[0088] The disclosed embodiments compress the target model to be deployed according to the hardware characteristics and application scenarios of the end-side device to obtain an inference function that adapts to the hardware characteristics and scenarios of the end-side device and can maintain a high level. Further based on the PowerInfer framework, heterogeneous computing is allocated to the processing tasks of the processor, and the interconnection pressure of the PCIe device is monitored. This effectively utilizes limited hardware resources and reduces the pressure on PCIe interconnection. The hybrid inference engine built under the PowerInfer framework is used to perceive the local memory usage of each processor's self-buffer, thereby further optimizing and adjusting the computing task allocation between the CPU and GPU, so that the overall performance of the end-side device is further optimized.

[0089] In addition, after optimizing the allocation of processing tasks based on memory usage efficiency, it also includes: obtaining test data for the deployment of backend-side device running tasks; if the test data does not meet the standards, optimize the deployment plan mentioned above based on the test data. Specifically, it includes optimization of lightweight target models, neuron-aware sparse operators, hybrid inference engines, and processing task allocation. Through continuous improvement, the deployment and operation efficiency of target models on end-side devices can be improved.

[0090] Combination Figure 5 As shown, the embodiment of the present disclosure provides a device model deployment apparatus 100, including a processor 101 and a memory 102. Optionally, the apparatus 100 may also include a communication interface 103 and a bus 104. The processor 101, the communication interface 103, and the memory 102 may communicate with each other through the bus 104. The communication interface 103 may be used for information transmission. The processor 101 may call the logic instructions in the memory 102 to execute the device model deployment method of the above embodiment.

[0091] In addition, the logic instructions in the memory 102 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

[0092] The memory 102 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 101 executes the functional application and data processing by running the program instructions / modules stored in the memory 102, that is, the device model deployment method in the above embodiment is implemented.

[0093] The memory 102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 102 may include a high-speed random access memory and may also include a non-volatile memory.

[0094] Combination Figure 6 As shown, an embodiment of the present disclosure provides an end-side device 200, including: a device body, and the above-mentioned device model deployment device 100. The device model deployment device 100 is installed on the device body. The installation relationship described here is not limited to placement inside the product body, but also includes installation connections with other components of the end-side device 200, including but not limited to physical connections, electrical connections, or signal transmission connections. It can be understood by those skilled in the art that the model deployment device 100 for the end-side device can be adapted to a feasible product body, thereby realizing other feasible embodiments.

[0095] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned device model deployment method.

[0096] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes.

[0097] The above description and the accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible changes. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and the claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0098] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods for each specific application to implement the described functions, but such implementations should not be considered to exceed the scope of the embodiments of the present disclosure. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0099] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0100] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A device model deployment method, characterized in that: include: According to the hardware characteristics and / or application scenarios of the end-side device, the model to be deployed is compressed to obtain a lightweight target model; Based on the model reasoning framework, the processing tasks of the lightweight target model are dynamically allocated; wherein the model reasoning framework and the lightweight target model are deployed on the terminal side device.

2. The device model deployment method according to claim 1, characterized in that: Based on the model reasoning framework, the processing tasks of the lightweight target model are dynamically allocated, including: Based on the model reasoning framework, the processing tasks of the lightweight target model are classified, and the processing tasks are assigned to the corresponding processors based on the classification; According to the perceived resource utilization of each processor, the task weight of the processor is dynamically adjusted to adjust the distribution of processing tasks.

3. The device model deployment method according to claim 1, characterized in that: The end-side device includes a graphics processor and a central processor connected through a data bus device; after dynamically allocating the processing tasks of the lightweight target model, it also includes: Obtaining interconnect stress on data bus devices when the CPU sends data to the GPU; According to the interconnection pressure, the bandwidth of sending data is adjusted; and the current task weight of each processor is adjusted.

4. The device model deployment method according to claim 3, characterized in that: Adjust the bandwidth of the data sent according to the interconnection pressure, including: When the interconnection pressure is less than the first pressure, adjusting the bandwidth of the data to be sent to a high-level bandwidth; When the interconnection pressure is greater than the second pressure, adjusting the bandwidth of the data to be sent to a lower level of bandwidth; The first pressure is lower than the second pressure.

5. The device model deployment method according to claim 1, characterized in that: According to the hardware characteristics and application scenarios of the end-side device, the target model to be deployed is compressed, including: Match the model compression processing solution in the preset database according to the hardware characteristics and application scenarios of the end-side device; The model to be deployed is compressed according to the matching compression processing scheme.

6. The device model deployment method according to any one of claims 1 to 5, characterized in that: Also includes: Call the hybrid reasoning engine built under the model reasoning framework; The hybrid inference engine is used to obtain the memory usage efficiency of the terminal device, and based on the memory usage efficiency, the allocation of processing tasks is optimized.

7. The device model deployment method according to claim 6, characterized in that: The hybrid inference engine under the model inference framework is constructed in the following way: Design neuron-aware sparse operators based on the network structure of the lightweight target model, input data, and hardware characteristics of the end-side device; According to the hardware capabilities and network requirements of the end-side device, a hybrid reasoning engine supporting the sparse operator is constructed and optimized under the model reasoning framework.

8. A model deployment device for a terminal device, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the device model deployment method according to any one of claims 1 to 7 when running the program instructions.

9. A terminal side device, characterized in that: include: End-side device body; The device model deployment apparatus as described in claim 8 is installed on the end-side device body.

10. A computer-readable storage medium storing program instructions, characterized in that: When the program instructions are executed, the computer is used to execute the device model deployment method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • An optimization method for deep learning of edge computing device

    CN110070181A

  • PCIe bandwidth real-time monitoring system and monitoring method thereof

    CN113722174A

  • Processing method and device

    CN114610328A

  • Neural network model compression method and device, equipment and medium

    CN114861910A

  • Network model training method and apparatus, and computer-readable storage medium

    WO2023071743A1

Cited By

  • Model compression method deployed on hardware equipment

    CN119808873A