Visual large model generation method and device of mobile terminal, storage medium and equipment
By employing differentiated pruning and SIMD structured sparse convolution on the visual Transformer model, the problem of imbalance between model accuracy and resource consumption in mobile deployments is solved, achieving efficient model compression and inference acceleration.
Patent Information
- Application Number
- CN202511057063.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-30
- Publication Date
- 2025-11-25
AI Technical Summary
When deploying visual Transformer models on mobile devices, existing pruning methods fail to effectively balance model accuracy and resource consumption, resulting in high computational costs and large memory usage, making it difficult to meet the actual needs of resource-constrained environments.
By employing a differentiated pruning strategy combined with SIMD structured sparse convolution, a pruning optimization function is constructed to optimize runtime by pruning the convolutional layers of large visual models. Furthermore, the SIMD instruction set and caching mechanism of modern CPUs are utilized to achieve efficient model compression.
While maintaining model accuracy, it significantly reduces memory usage and computational overhead, improves the model's inference speed and adaptability on mobile devices, and enhances its usability on resource-constrained devices.
Smart Images
Figure CN121009984A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for generating large visual models for mobile devices, a storage medium, and computer equipment. Background Technology
[0002] In recent years, the Visual Transformer (ViT) has made groundbreaking progress in the field of computer vision, surpassing many traditional models and becoming a hot research topic. However, in practical applications, especially in mobile inference, its outstanding performance is underpinned by a massive parameter set of hundreds of millions, leading to serious memory and computational cost issues. In industrial settings and other environments with stringent requirements for efficiency and resource management, the large-scale application of ViT faces significant challenges, urgently necessitating efficient model compression methods to specifically improve the models for practical applications.
[0003] In the field of neural network compression, network pruning is a proven effective method for convolutional neural networks (CNNs). Early research mainly focused on theoretically reducing model parameters and floating-point operations. However, in actual network deployments, it was found that model runtime is not strictly dependent on computational cost and parameter count, but rather influenced by a combination of factors. Therefore, some research has shifted to resource-constrained network compression, incorporating device power consumption into the optimization objective. This has achieved a balance between runtime, power consumption, and network accuracy to some extent. However, existing algorithms do not fully consider the target deployment devices when designing pruning patterns, so there is still room for performance improvement.
[0004] In CNN pruning research, while unstructured pruning can remove a large number of parameters without sacrificing accuracy, its random memory access mode is incompatible with modern CPU SIMD technology and cache structures, resulting in poor practical acceleration. Secondly, although structured pruning is hardware-friendly and can achieve ideal acceleration, its strong channel-level constraints make it difficult to achieve high compression rates and speedups while maintaining accuracy. In summary, when designing network pruning algorithms, it is crucial to fully consider the computing architecture of the target deployment device, especially for ViT applications on mobile devices, where new pruning strategies need to be explored to achieve a better balance between accuracy and efficiency. Summary of the Invention
[0005] In view of this, embodiments of this application provide a method and apparatus for generating large visual models on a mobile device, a storage medium, and a computer device.
[0006] According to one aspect of this application, a method for generating large visual models on a mobile device is provided, the method comprising:
[0007] Pruning operations with different strategies are performed on each convolutional layer in the large visual model to determine the parameters of each convolutional layer with different numbers of parameters.
[0008] SIMD structured sparse convolution is performed on the parameters of each convolutional layer with different numbers of parameters to determine the computation time for each convolutional layer with different numbers of parameters.
[0009] Using the runtime of the large visual model as a constraint, a pruning optimization function for the large visual model is constructed. Based on the computation time of different numbers of parameters corresponding to each convolutional layer, the pruning optimization function is solved to determine the pruning strategy for the large visual model.
[0010] The visual model is pruned according to the pruning strategy to obtain the visual model for the mobile device.
[0011] According to another aspect of this application, a mobile visual large model generation device is provided, the device comprising:
[0012] The pruning module is used to perform pruning operations on each convolutional layer in the large visual model using different strategies, and to determine the parameters of each convolutional layer with different numbers of parameters.
[0013] The computation module is used to perform SIMD structured sparse convolution computation on the parameters of each convolutional layer with different numbers of parameters, and to determine the computation time for each convolutional layer with different numbers of parameters.
[0014] The optimization module is used to construct a pruning optimization function for the large visual model with the running time of the large visual model as a constraint, and solve the pruning optimization function according to the computation time of different numbers of parameters corresponding to each convolutional layer to determine the pruning strategy for the large visual model.
[0015] The pruning module is also used to perform pruning operations on the large visual model according to the pruning strategy to obtain the large visual model for the mobile device.
[0016] According to another aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described method for generating large visual models for mobile devices.
[0017] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described method for generating large visual models for a mobile terminal.
[0018] By employing the above technical solutions, the present application provides a method, apparatus, storage medium, and computer device for generating large visual models on mobile devices. Through differentiated pruning strategies applied to different convolutional layers, it retains the expressive power of key features while effectively removing redundant parameters, thereby achieving a high compression ratio while maintaining model accuracy. Combined with the implementation of structured sparse convolution, the pruned model can fully utilize the SIMD instruction set and caching mechanism of modern CPUs, overcoming the inefficiency of traditional unstructured pruning in practical deployment and improving inference speed. Furthermore, the pruning strategy modeling method, with runtime optimization as the objective, makes the entire compression process closer to the needs of real-world application scenarios, avoiding the disconnect between theoretical metrics and actual performance, and improving the adaptability and stability of the model for mobile deployment. Overall, the present application, while ensuring accuracy, can reduce model memory usage and computational overhead, enhancing the practicality and scalability of large visual models on resource-constrained devices.
[0019] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0021] Figure 1 A flowchart illustrating a method for generating large visual models on a mobile device according to an embodiment of this application is shown.
[0022] Figure 2 This paper illustrates a flowchart of another method for generating large visual models on a mobile device, as provided in an embodiment of this application.
[0023] Figure 3 A schematic diagram of the structure of a mobile visual large model generation device provided in an embodiment of this application is shown. Detailed Implementation
[0024] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.
[0025] This embodiment provides a method for generating large visual models on mobile devices, such as... Figure 1 As shown, the method includes:
[0026] Step 101: Perform pruning operations on each convolutional layer in the large visual model using different strategies to determine the convolutional layer parameters with different numbers of parameters for each convolutional layer.
[0027] Step 102: Perform SIMD structured sparse convolution calculations on the parameters of each convolutional layer with different numbers of parameters to determine the computation time for each convolutional layer with different numbers of parameters.
[0028] Step 103: Using the runtime of the large visual model as a constraint, construct a pruning optimization function for the large visual model, and solve the pruning optimization function according to the computation time of different numbers of parameters corresponding to each convolutional layer to determine the pruning strategy for the large visual model.
[0029] Step 104: Perform pruning operations on the large visual model according to the pruning strategy to obtain the large visual model for the mobile device.
[0030] This application proposes an efficient model compression method for Visual Transformer (ViT) deployed on mobile devices. The core of this method combines the advantages of unstructured pruning and structured sparse computation. By designing a pruning strategy optimized for actual runtime, it improves inference efficiency and reduces resource consumption while maintaining model accuracy. First, different pruning strategies are adopted based on the characteristics and importance of different convolutional layers. Multiple pruning strategies are used for each convolutional layer to obtain convolutional layer parameters with different numbers of parameters. Second, the pruned sparse structure is transformed into a form that can be efficiently executed on modern CPU architectures. Although unstructured pruning can remove a large number of redundant parameters, its random access mode is difficult to accelerate using SIMD instruction sets. Therefore, this solution adapts to hardware characteristics through a structured sparse convolution implementation, thereby truly accelerating inference. Furthermore, unlike traditional pruning methods that only target the number of parameters or FLOPs, this application establishes a mathematical optimization problem with runtime as a constraint, and models and solves the initial large-scale visual model based on the measured computation time under different pruning strategies. This allows for the automatic search for the optimal pruning strategy that balances accuracy, compression ratio, and inference speed on a given device platform. Finally, the original model is pruned according to the optimized pruning strategy, resulting in a lightweight model that runs efficiently on a specific mobile device with controllable accuracy loss. This process includes not only adjustments to the model structure but also optimization of the underlying computational logic, ensuring that the compressed model can achieve optimal performance in actual deployment.
[0031] By applying the technical solution of this embodiment, and implementing differentiated pruning strategies for different convolutional layers, key feature representation capabilities are preserved while redundant parameters are effectively removed, thus achieving a high compression ratio while ensuring model accuracy. Combined with the implementation of structured sparse convolution, the pruned model can fully utilize the SIMD instruction set and caching mechanism of modern CPUs, overcoming the inefficiency of traditional unstructured pruning in practical deployment and improving inference speed. Furthermore, the pruning strategy modeling method, with runtime optimization as the objective, makes the entire compression process closer to the needs of real-world application scenarios, avoiding the disconnect between theoretical metrics and actual performance, and improving the adaptability and stability of the model for mobile deployment. Overall, this embodiment, while ensuring accuracy, can reduce model memory usage and computational overhead, enhancing the practicality and scalability of large-scale visual models on resource-constrained devices.
[0032] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, and to fully illustrate the specific implementation process of this embodiment, another method for generating large visual models on mobile devices is provided, such as... Figure 2 As shown, the method includes:
[0033] Step 201: Obtain the length of the vector register in the CPU; based on the length of the vector register, group the output channel parameters of each convolutional layer in the large visual model so that the number of output channel parameters in each group matches the length of the vector register; write the output channel parameters of each group into the corresponding vector register; perform pruning operations with different strategies on each convolutional layer in the grouped large visual model to determine the convolutional layer parameters with different numbers of parameters corresponding to each convolutional layer, so that the output channel parameters in the same vector register are simultaneously retained or deleted.
[0034] In this embodiment, modern CPUs generally support SIMD (Single Instruction Multiple Data) instruction sets, such as ARM NEON and x86 SSE / AVX. These instructions process multiple data elements in parallel through vector registers, thereby accelerating computation tasks. The length of the vector register determines the number of data that can be processed in parallel at a time. For example, in mobile CPUs supporting NEON, the vector register is usually 128 bits. If FP32 precision is used, it can process four floating-point numbers at a time. Obtaining this parameter is to consider the parallel capabilities of the underlying hardware during pruning, ensuring that the pruned model structure can be executed efficiently. SIMD technology is an important means for modern CPUs to achieve data parallelism. It places a set of vector registers that can load multiple operands simultaneously inside the CPU chip and sets up multiple logic units that can operate on this data simultaneously, enabling the CPU to operate on multiple data simultaneously using only one arithmetic instruction, thus improving its data parallelism capability. For example, if the vector register length is 4, the output channels are divided into groups of four. In SIMD structured pruning, network parameters are grouped according to the length of the vector registers in the CPU, ensuring that parameters in the same group are removed or retained simultaneously. In practical implementation, let the convolution parameters of the l-th layer of the network be... Where i c O c k h k w These are the number of input / output channels of the convolutional layer, and the height and width of the convolutional kernel, with parameters along dimension O. c The parameters are grouped, with the number of parameters in each group related to the length of the vector registers in the CPU. This allows for a better semi-structured pruning method between structured and unstructured pruning, considering both the hardware's parallel computing capabilities and avoiding the accuracy loss caused by excessive structuring. After mapping each group of output channel parameters to the corresponding vector register, when performing pruning operations on each group, all channel parameters within the same vector register must either be completely retained or completely deleted. By aligning the pruning granularity with the vector register length, the pruned model can fully utilize the CPU's parallel computing capabilities during execution, significantly improving inference speed. Simultaneously, the design of forcing the complete retention or deletion of parameters within the same vector register avoids the accuracy degradation caused by strong constraints in traditional structured pruning, achieving a balance between greater flexibility and compression ratio.
[0035] Step 202: Based on the output channel parameters in the vector registers corresponding to each convolutional layer, determine the sparse matrix and save the column number, column index, and non-zero column values of the sparse matrix; obtain the input dense matrix corresponding to each convolutional layer, obtain the corresponding column values in the input dense matrix based on the column index of the non-zero column of the corresponding convolutional layer, and perform SIMD structured sparse convolution calculation on the non-zero column values in the sparse matrix and the corresponding column values in the input dense matrix to determine the calculation results and calculation time for different numbers of parameters corresponding to each convolutional layer.
[0036] In this embodiment, after the pruning operation, a large number of zero-value parameters will appear in the weight matrix of each convolutional layer. To avoid invalid operations and improve execution efficiency in subsequent calculations, this step first constructs a sparse matrix based on the output channel parameters retained by the current convolutional layer (i.e., the pruned parameters), and then extracts the following from it: the number of non-zero columns: representing the number of output channels actually participating in the effective calculation; the column index of the non-zero column: used to locate the corresponding position in the original input feature map that needs to participate in the calculation; and the value of the non-zero column: i.e., the retained convolutional kernel parameter values. This sparse matrix representation method not only reduces storage overhead but also provides a foundation for efficient subsequent calculations. Next, when performing convolution calculations, in addition to the sparse convolutional kernel parameters, an input feature map is also required as input data. Since the input is usually dense data (such as images or intermediate features), the corresponding dense input matrix is directly obtained, and the corresponding column values are extracted from the input based on the non-zero column index obtained in the previous step. This ensures that only those input data related to the retained convolutional kernel parameters are calculated, thereby avoiding redundant operations and improving computational efficiency. Finally, convolution is performed between the non-zero column values of the sparse matrix and the corresponding column values of the input dense matrix. Since this data has been organized into a vector register-friendly format, and the size of each group matches the length of the CPU's vector register, multiple groups of data can be processed in parallel using the SIMD instruction set, improving computation speed. During this process, computation is performed on a mobile device, and the computation results and time for each layer are recorded, providing empirical evidence for the subsequent construction of pruning optimization functions. This approach utilizes sparse matrix compression technology to reduce the computational burden and memory consumption caused by redundant parameters. Simultaneously, the non-zero column indexing mechanism accurately matches the input data, avoiding invalid computations. Furthermore, by combining the SIMD architecture characteristics of modern CPUs, the pruned convolution operation is transformed into efficient vectorized computation, fully leveraging the parallel processing capabilities of hardware resources, thereby improving inference speed. In addition, this method dynamically collects the computation time of each layer during runtime, providing real performance feedback for subsequent pruning strategy optimization, making the entire model compression process more closely reflect the needs of actual deployment environments.
[0037] Step 203: With the constraint that the running time of the large visual model does not exceed the preset maximum acceptable running time, construct a pruning optimization function with the objective of minimizing the loss of the large visual model. The pruning optimization function is expressed as follows: stT(W)≤T budget Where W represents the parameters of the convolutional layer in the large visual model, L(W) represents the loss function of the large visual model, and T(W) is the running time of the convolutional layer in the large visual model on a mobile device when the parameters are W. budget This is the preset maximum acceptable runtime.
[0038] Step 204: Solve the pruning optimization function based on the computation time of different numbers of parameters corresponding to each convolutional layer to determine the pruning strategy for the large visual model.
[0039] In this embodiment of the application, optionally, the running time of each convolutional layer in the large visual model on a mobile device when the output channel parameter is W is determined by interpolation calculation based on the following formula for the computation time of different numbers of parameters: in, This represents the computation time of convolutional layer l when the number of output channel parameters in convolutional layer l is s. This represents the computation time corresponding to the first sparse matrix of convolutional layer l. This represents the computation time corresponding to the i-th sparse matrix of convolutional layer l. This represents the number of output channel parameters corresponding to the i-th sparse matrix of convolutional layer l; the order of the sparse matrices is obtained by arranging the output channel parameters corresponding to convolutional layer l in ascending order, and n is the number of sparse matrices.
[0040] In this embodiment, for sparse convolution, the runtime of a specific convolutional layer is determined by the number of parameters retained in that layer. Overall, the runtime of sparse convolution does not have a standard linear relationship with the number of non-zero parameters in the convolutional layer. However, when the number of non-zero parameters varies within a small range, the computation time of sparse convolution becomes approximately linearly related to the number of non-zero parameters. Based on this approximately linear relationship, this embodiment proposes using linear interpolation to achieve a fast estimation of the runtime of sparse convolution. Specifically, for the l-th convolutional layer in the network, the number of non-zero parameters is first measured to different values. At that time, the actual runtime of sparse convolution on the device. Then, according to the linear interpolation formula... Calculate the estimated runtime of sparse convolution when the number of convolution parameters (number of output channel parameters) is s. It is an indicator variable. When the number of non-zero parameters in the convolutional layer is and When the time between these parameters varies, the slope between the runtime of sparse convolution and the number of non-zero parameters in the convolutional layer is:
[0041] In this embodiment of the application, optionally, the step of solving the pruning optimization function based on the computation time of different numbers of parameters corresponding to each convolutional layer to determine the pruning strategy for the large visual model includes: introducing auxiliary variables Z and Lagrange multipliers λ to construct augmented Lagrange function pairs: Where ρ represents the penalty parameter; by taking the partial derivative of the augmented Lagrangian function with respect to W and setting it equal to 0, the update formula for W is obtained; by taking the partial derivative of the augmented Lagrangian function with respect to Z and setting it equal to 0, the update formula for Z is obtained; according to λ new =λ+ρ(T(Z)-T budget Update the Lagrange multiplier λ, where λ new Represent the new Lagrange multiplier, and repeat the above steps until the maximum number of iterations is reached or the error range is less than the set threshold, at which point the iteration stops and the final optimized parameter W is obtained.
[0042] In this embodiment, eliminating model parameters provides the necessary basis for subsequent optimization using runtime as a constraint, ensuring that the pruned network meets runtime requirements. Specifically:
[0043] Introducing auxiliary variable Z and Lagrange multiplier λ, the constraint T(W)≤T budget Transform into an equality constraint T(Z) = T budget In this process, Z gradually approaches the optimal W. Construct the augmented Lagrangian function: Where ρ is the penalty parameter, used to control the severity of the penalty for constraint violation. The solution is obtained through alternating iterations: first, the partial derivative of the augmented Lagrangian function with respect to W is taken and set to 0 to obtain the update formula for W; second, the partial derivative of the augmented Lagrangian function with respect to Z is taken and set to 0 to obtain the update formula for Z; finally, based on λ... new =λ+ρ(T(Z)-T budget Update the Lagrange multiplier λ, where λ new This represents the new Lagrange multiplier. Repeat the above steps until the maximum number of iterations is reached or the error range is less than the set threshold, then stop the iteration and obtain the final optimized network parameters W.
[0044] Step 205: Prune the large visual model according to the pruning strategy to obtain a large visual model for the mobile device; deploy the large visual model on the mobile device to perform visual image processing.
[0045] In this embodiment, the initial large visual model is pruned according to the optimal pruning strategy obtained by solving, and the pruned large visual model is deployed on the mobile device, thereby using the large visual model on the mobile device for visual image processing.
[0046] Compared with the prior art, the advantages of applying the technical solution of this embodiment include:
[0047] 1. By using a semi-structured pruning mode, the computational load and memory requirements of the model are reduced, while the accuracy loss caused by excessive structured or unstructured pruning is effectively avoided, achieving a good balance between model accuracy and computational efficiency.
[0048] 2. The SIMD structured pruning mode and related computational algorithms fully consider the hardware architecture characteristics of mobile CPUs, enabling the pruned model to make full use of SIMD technology for efficient computation, reducing the randomness of data access, and improving cache utilization. Compared with unstructured pruning, it can achieve a more ideal acceleration effect in actual deployment, effectively improving the running efficiency of the model on mobile devices, and providing strong support for the widespread application of large models on edge computing devices.
[0049] 3. By incorporating the runtime of the pruned network on a real device into the constraints and estimating it using an efficient linear interpolation method, the network runtime can be precisely controlled, ensuring that the real-time requirements of practical applications are met, thus enhancing the applicability and practicality of the algorithm.
[0050] Furthermore, as Figure 1 In terms of specific implementation, this application provides a mobile-based visual large model generation device, such as... Figure 3 As shown, the device includes:
[0051] The pruning module is used to perform pruning operations on each convolutional layer in the large visual model using different strategies, and to determine the parameters of each convolutional layer with different numbers of parameters.
[0052] The computation module is used to perform SIMD structured sparse convolution computation on the parameters of each convolutional layer with different numbers of parameters, and to determine the computation time for each convolutional layer with different numbers of parameters.
[0053] The optimization module is used to construct a pruning optimization function for the large visual model with the running time of the large visual model as a constraint, and solve the pruning optimization function according to the computation time of different numbers of parameters corresponding to each convolutional layer to determine the pruning strategy for the large visual model.
[0054] The pruning module is also used to perform pruning operations on the large visual model according to the pruning strategy to obtain the large visual model for the mobile device.
[0055] Optionally, in this embodiment of the application, the pruning module is used for:
[0056] Get the length of the vector register in the CPU;
[0057] Based on the length of the vector register, the output channel parameters of each convolutional layer in the large visual model are grouped so that the number of output channel parameters in each group matches the length of the vector register.
[0058] The output channel parameters of each group are written into the corresponding vector register. Different pruning strategies are applied to each convolutional layer in the grouped visual large model to determine the convolutional layer parameters with different numbers of parameters corresponding to each convolutional layer, so that the output channel parameters in the same vector register are simultaneously retained or deleted.
[0059] Optionally, in this embodiment of the application, the computing module is used for:
[0060] Based on the output channel parameters in the vector registers corresponding to each convolutional layer, determine the sparse matrix and save the column number, column index, and non-zero column values of the sparse matrix.
[0061] Obtain the input dense matrix corresponding to each convolutional layer, obtain the corresponding column value in the input dense matrix based on the column index of the non-zero column of the corresponding convolutional layer, and perform SIMD structured sparse convolution calculation on the non-zero column value in the sparse matrix and the corresponding column value in the input dense matrix to determine the calculation results and calculation time for different numbers of parameters corresponding to each convolutional layer.
[0062] Optionally, in this embodiment of the application, the optimization module is used to:
[0063] With the constraint that the running time of the large visual model does not exceed a preset maximum acceptable running time, a pruning optimization function is constructed with the objective of minimizing the loss of the large visual model. The pruning optimization function is expressed as follows:
[0064] stT(W)≤T budget Where W represents the parameters of the convolutional layer in the large visual model, L(W) represents the loss function of the large visual model, and T(W) is the running time of the convolutional layer in the large visual model on a mobile device when the parameters are W. budget This is the preset maximum acceptable runtime.
[0065] In this embodiment of the application, optionally, the running time of each convolutional layer in the large visual model on a mobile device when the output channel parameter is W is determined by interpolation calculation based on the following formula for the computation time of different numbers of parameters:
[0066]
[0067] in, This represents the computation time of convolutional layer l when the number of output channel parameters in convolutional layer l is s. This represents the computation time corresponding to the first sparse matrix of convolutional layer l. This represents the computation time corresponding to the i-th sparse matrix of convolutional layer l. This represents the number of output channel parameters corresponding to the i-th sparse matrix of convolutional layer l; the order of the sparse matrices is obtained by arranging the output channel parameters corresponding to convolutional layer l in ascending order, and n is the number of sparse matrices.
[0068] Optionally, in this embodiment of the application, the optimization module is used to:
[0069] By introducing auxiliary variable Z and Lagrange multiplier λ, we construct an augmented Lagrange function pair:
[0070] Where ρ represents the penalty parameter;
[0071] Taking the partial derivative of the augmented Lagrangian function with respect to W and setting it equal to 0, we obtain the updated formula for W; taking the partial derivative of the augmented Lagrangian function with respect to Z and setting it equal to 0, we obtain the updated formula for Z; according to λ new =λ+ρ(T(Z)-T budget Update the Lagrange multiplier λ, where λ new Represent the new Lagrange multiplier, and repeat the above steps until the maximum number of iterations is reached or the error range is less than the set threshold, at which point the iteration stops and the final optimized parameter W is obtained.
[0072] Optionally, in this embodiment, the pruning module is further configured to deploy the large visual model on a mobile device to perform visual image processing using the large visual model deployed on the mobile device.
[0073] It should be noted that other corresponding descriptions of the functional units involved in the mobile visual large model generation device provided in this application embodiment can be found in the following references. Figures 1 to 2 The corresponding descriptions in the method will not be repeated here.
[0074] This application also provides a computer device, specifically a personal computer, server, network device, etc. The computer device includes a bus, processor, memory, and communication interface, and may also include input / output interfaces and a display device. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores location information. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.
[0075] Those skilled in the art will understand that the structure of the computer device described above is only a partial structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. A specific computer device may include more or fewer components, or combine certain components, or have different component arrangements.
[0076] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0077] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0078] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0079] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0080] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0081] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for generating large visual models on a mobile device, characterized in that, The method includes: Pruning operations with different strategies are performed on each convolutional layer in the large visual model to determine the parameters of each convolutional layer with different numbers of parameters. SIMD structured sparse convolution is performed on the parameters of each convolutional layer with different numbers of parameters to determine the computation time for each convolutional layer with different numbers of parameters. Using the runtime of the large visual model as a constraint, a pruning optimization function for the large visual model is constructed. Based on the computation time of different numbers of parameters corresponding to each convolutional layer, the pruning optimization function is solved to determine the pruning strategy for the large visual model. The visual model is pruned according to the pruning strategy to obtain the visual model for the mobile device.
2. The method according to claim 1, characterized in that, The process of performing pruning operations with different strategies on each convolutional layer in the large visual model to determine the convolutional layer parameters with different numbers of parameters for each convolutional layer includes: Get the length of the vector register in the CPU; Based on the length of the vector register, the output channel parameters of each convolutional layer in the large visual model are grouped so that the number of output channel parameters in each group matches the length of the vector register. The output channel parameters of each group are written into the corresponding vector register. Different pruning strategies are applied to each convolutional layer in the grouped visual large model to determine the convolutional layer parameters with different numbers of parameters corresponding to each convolutional layer, so that the output channel parameters in the same vector register are simultaneously retained or deleted.
3. The method according to claim 2, characterized in that, The step of performing SIMD structured sparse convolution calculations on the parameters of each convolutional layer with different numbers of parameters, and determining the computation time for each convolutional layer with different numbers of parameters, includes: Based on the output channel parameters in the vector registers corresponding to each convolutional layer, determine the sparse matrix and save the column number, column index, and non-zero column values of the sparse matrix. Obtain the input dense matrix corresponding to each convolutional layer, obtain the corresponding column value in the input dense matrix based on the column index of the non-zero column of the corresponding convolutional layer, and perform SIMD structured sparse convolution calculation on the non-zero column value in the sparse matrix and the corresponding column value in the input dense matrix to determine the calculation results and calculation time for different numbers of parameters corresponding to each convolutional layer.
4. The method according to claim 1, characterized in that, The pruning optimization function for the large visual model, which is constructed with the runtime of the large visual model as a constraint, includes: With the constraint that the running time of the large visual model does not exceed a preset maximum acceptable running time, a pruning optimization function is constructed with the objective of minimizing the loss of the large visual model. The pruning optimization function is expressed as follows: stT(W)≤T budget Where W represents the parameters of the convolutional layer in the large visual model, L(W) represents the loss function of the large visual model, and T(W) is the running time of the convolutional layer in the large visual model on a mobile device when the parameters are W. budget This is the preset maximum acceptable runtime.
5. The method according to claim 4, characterized in that, The runtime of each convolutional layer in the large visual model on a mobile device with output channel parameters of W is determined by interpolation calculation based on the following formula for different numbers of parameters: in, This represents the computation time of convolutional layer l when the number of output channel parameters in convolutional layer l is s. This represents the computation time corresponding to the first sparse matrix of convolutional layer l. This represents the computation time corresponding to the i-th sparse matrix of convolutional layer l. This represents the number of output channel parameters corresponding to the i-th sparse matrix of convolutional layer l; the order of the sparse matrices is obtained by arranging the output channel parameters corresponding to convolutional layer l in ascending order, and n is the number of sparse matrices.
6. The method according to claim 5, characterized in that, The step of determining the pruning strategy for the large visual model by solving the pruning optimization function based on the computation time of different numbers of parameters corresponding to each convolutional layer includes: By introducing auxiliary variable Z and Lagrange multiplier λ, we construct an augmented Lagrange function pair: Where ρ represents the penalty parameter; Taking the partial derivative of the augmented Lagrangian function with respect to W and setting it equal to 0, we obtain the updated formula for W; taking the partial derivative of the augmented Lagrangian function with respect to Z and setting it equal to 0, we obtain the updated formula for Z; according to λ new =λ+ρ(T(Z)-T budget Update the Lagrange multiplier λ, where λ new Represent the new Lagrange multiplier, and repeat the above steps until the maximum number of iterations is reached or the error range is less than the set threshold, at which point the iteration stops and the final optimized parameter W is obtained.
7. The method according to any one of claims 1 to 6, characterized in that, After pruning the large visual model according to the pruning strategy to obtain the large visual model for the mobile device, the method further includes: Deploy large visual models on mobile devices to perform visual image processing using large visual models deployed on mobile devices.
8. A mobile-based visual large-scale model generation device, characterized in that, The device includes: The pruning module is used to perform pruning operations on each convolutional layer in the large visual model using different strategies, and to determine the parameters of each convolutional layer with different numbers of parameters. The computation module is used to perform SIMD structured sparse convolution computation on the parameters of each convolutional layer with different numbers of parameters, and to determine the computation time for each convolutional layer with different numbers of parameters. The optimization module is used to construct a pruning optimization function for the large visual model with the running time of the large visual model as a constraint, and solve the pruning optimization function according to the computation time of different numbers of parameters corresponding to each convolutional layer to determine the pruning strategy for the large visual model. The pruning module is also used to perform pruning operations on the large visual model according to the pruning strategy to obtain the large visual model for the mobile device.
9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.
10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.