Neural network model deployment method and system based on cloud collaboration

Abstract

The application discloses a neural network model deployment method and system based on cloud cooperation, and belongs to the technical field of edge computing. The method comprises the following steps: in response to the selection of a user, loading an original model and configured compilation parameters into a main container, and matching a corresponding sub-container; the sub-container performs a compilation task according to the compilation parameters, obtains an offline neural network model, performs evaluation analysis, determines a performance evaluation result, and returns the performance evaluation result and the offline neural network model to the main container; the main container selects an optimal edge device according to the performance evaluation result, and deploys the offline neural network model to the optimal edge device; and the optimal edge device runs the offline neural network model to perform an inference task, and returns an inference result to the main container. The method can quickly compile and generate offline inference models for different hardware platforms, is more efficient in application to a production environment, and solves the problem of low model compilation efficiency under inference devices of various architectures.

Description

Technical Field

[0001] This invention relates to the field of edge computing technology, and in particular to a method and system for deploying a neural network model based on cloud collaboration. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] With the increasing maturity of deep learning technology, the application of artificial intelligence based on deep learning and neural networks has received widespread attention from academia and industry, especially in the field of edge computing.

[0004] The goal of edge computing is to provide computing, storage, and network bandwidth closer to data input or users, thereby achieving more efficient inference responses. Edge computing involves generating and deploying offline neural networks, loading models onto edge devices for inference. However, in the process of deploying AI applications at the edge, the limited computing and memory resources of the inference environment, coupled with the diverse range of devices available for edge inference and the need to select the optimal deployment solution, necessitate the sequential verification of different models' actual inference performance across different frameworks and platforms. Therefore, in actual production, the offline model conversion and multi-platform offline model deployment significantly impact application efficiency.

[0005] In the offline model conversion stage, the host device needs to configure the environment for the compiler. The configuration methods differ across platforms, and the parameters or files required vary depending on the deep learning framework. Similarly, directly validating or applying the compiled offline model to various types of edge devices without preliminary performance analysis will result in a significant amount of manual configuration and model conversion work, making it difficult to rapidly generate offline models in batches.

[0006] In the offline model deployment phase, due to the different architectures of different platforms, targeted reconfiguration is required for each hardware. For a single piece of hardware, a one-time configuration can provide long-term inference support for that hardware. However, this method is inefficient when dealing with multiple hardware platforms. Similarly, traditional deployment methods involve a single compilation and manual deployment, which not only fails to verify the actual performance of model inference in real time, but also requires recompilation and deployment if performance metrics are not met, resulting in low efficiency.

[0007] While compilation and deployment for a single scenario are relatively mature, manually compiling and deploying repeatedly for inference devices with various architectures and model framework platforms is not only inefficient but also involves complex configuration environments. Different hosts can lead to different errors, resulting in extremely low work efficiency. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a cloud-based collaborative neural network model deployment method and system. Utilizing the isolation features of Docker (containers), it shields users from different environmental characteristics and improves the model compilation and deployment process. This allows users to interactively send tasks to different compilation environments in a single go, enabling compilation tasks across multiple platforms. Performance analysis of the compilation results is then performed to recommend optimal improvement and implementation schemes. The returned offline neural network model is then sent to edge devices for inference tasks, enabling more efficient application in production environments.

[0009] In a first aspect, the present invention provides a method for deploying a neural network model based on cloud-based collaboration;

[0010] A cloud-based collaborative neural network model deployment method includes:

[0011] In response to the user's selection, the original model and the configured compilation parameters are loaded into the main container and matched with the corresponding sub-containers;

[0012] The sub-container executes the compilation task according to the compilation parameters, obtains the offline neural network model and performs evaluation and analysis, determines the performance evaluation result, and returns the performance evaluation result and the offline neural network model to the main container;

[0013] In response to the optimal edge device selected by the user based on performance evaluation results, the main container deploys the offline neural network model to the optimal edge device;

[0014] The optimal edge device runs an offline neural network model to perform inference tasks and returns the inference results to the main container.

[0015] Furthermore, the evaluation and analysis of offline neural network models specifically includes:

[0016] The total execution time of the model inference is calculated by measuring the memory access time, computation time, and overlap time during the offline neural network model runtime.

[0017] Determine whether the total execution time of the model inference meets the preset production demand indicators;

[0018] If not, calculate the arithmetic strength and operation byte ratio of the offline neural network model, and determine the restricted type of the offline neural network model based on the arithmetic strength and operation byte ratio;

[0019] Based on the type of constraint, optimization suggestions are generated to help users modify the model.

[0020] Preferably, the arithmetic strength and operation byte ratio of the offline neural network model are specifically as follows:

[0021] The arithmetic strength of the offline neural network model is obtained based on the computation time, number of operations, and processor bandwidth of the offline neural network model.

[0022] Based on the processor's computing bandwidth and memory bandwidth, obtain the operation byte ratio of the offline neural network model.

[0023] Preferably, determining the restricted type of the offline neural network model based on arithmetic strength and operand-to-byte ratio includes:

[0024] When the arithmetic strength is greater than the operand ratio, and the difference between the arithmetic strength and the operand ratio is greater than a preset threshold, the offline neural network model is classified as computationally constrained.

[0025] When the arithmetic strength is less than the operand ratio, and the difference between the arithmetic strength and the operand ratio is greater than a preset threshold, the offline neural network model is classified as memory-constrained.

[0026] Otherwise, the constraint type of the offline neural network model is delay-constrained.

[0027] Preferably, generating optimization suggestions based on the constraint type includes:

[0028] If the constraint type is computationally constrained, the optimization suggestions include model compression, model optimization, distributed computing, and hardware optimization.

[0029] If the constraint type is memory-constrained, optimization suggestions include batch processing, streaming data processing, optimizing data structures, and replanning memory reclamation and management;

[0030] If the constraint type is latency-constrained, optimization suggestions include asynchronous computing, cloud-edge collaboration, and memory latency hiding.

[0031] Furthermore, there are multiple sub-containers, each corresponding to a different hardware carrier platform.

[0032] Furthermore, the main container sends compilation parameters to the child container while simultaneously creating the first and second shared folders;

[0033] The child container shares the original model to be compiled with the child container through the first shared folder, and the main container obtains the offline neural network model and performance analysis results returned by the child container through the second shared folder;

[0034] The child container listens to the first and second shared folders.

[0035] Furthermore, if the performance evaluation results are unsatisfactory, the following steps are taken before selecting the optimal edge device:

[0036] In response to user selection, update the original model and / or configuration parameters to allow the sub-container to correct the offline neural network model until the performance evaluation results meet the requirements.

[0037] Secondly, the present invention provides a cloud-based collaborative neural network model deployment system;

[0038] A cloud-based collaborative neural network model deployment system, comprising a server and optimal edge devices;

[0039] The server responds to the user's selection by loading the original model and configured compilation parameters into the main container and matching the corresponding sub-container. The sub-container executes the compilation task according to the compilation parameters, obtains the offline neural network model, performs evaluation and analysis, determines the performance evaluation result, and returns the performance evaluation result and the offline neural network model to the main container. The main container selects the optimal edge device based on the performance evaluation result and deploys the offline neural network model to the optimal edge device.

[0040] The optimal edge device is used to run offline neural network models to perform inference tasks and return the inference results to the main container.

[0041] Furthermore, the optimal edge device includes a data acquisition module, a host, and an edge module;

[0042] The offline neural network model is deployed to the edge module via the host. The data acquisition module collects data and transmits it to the edge module. The edge module runs the offline neural network model to process the data, performs inference tasks, and returns the inference results to the main container.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] 1. The technical solution provided by this invention utilizes the isolation features of Docker to shield different environmental characteristics, while improving the compilation and deployment process. This allows users to send tasks to different compilation environments in an interactive manner, enabling compilation tasks for multiple platforms. At the same time, the compilation results are analyzed for performance, and the best improvement and implementation schemes are recommended. Finally, the returned offline neural network model is sent to edge devices to perform inference tasks, so as to be more efficiently applied to the production environment.

[0045] 2. The technical solution provided by this invention, based on basic throughput and latency analysis, introduces arithmetic strength and operation byte ratio to evaluate and analyze the neural network model. In actual inference, it can provide optimization directions based on the evaluation results, and optimize the hardware to maximize hardware performance.

[0046] 3. The technical solution provided by this invention eliminates 70% of the manual configuration operations after initialization. Users only need to perform one operation to obtain the inference files required by different hardware and to execute inference tasks on the host machine after adjusting the environment. After system initialization, the manual operation process and steps are significantly reduced in the execution flow, resulting in a multiple-fold improvement in execution efficiency. Attached Figure Description

[0047] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0048] Figure 1 This is a flowchart provided for an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of the hardware framework provided in an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of the software framework provided in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the analysis and optimization recommendation process provided in the embodiments of the present invention;

[0052] Figure 5 This is a schematic diagram of the execution flow provided in an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of the execution flow of the prior art provided for embodiments of the present invention. Detailed Implementation

[0054] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0055] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0056] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0057] Example 1

[0058] Existing model compilation and deployment methods suffer from low execution efficiency and are prone to errors when dealing with inference devices with complex configuration environments and various model framework platforms. Therefore, this invention provides a cloud-based collaborative neural network model deployment method that enables a single configuration to distribute and execute tasks multiple times on different hardware, allowing for efficient reuse.

[0059] Next, combined Figures 1-6 This embodiment discloses a cloud-based collaborative neural network model deployment method, which includes the following steps:

[0060] S1. In response to the user's selection, the original model and configured compilation parameters are loaded into the main container and matched with the corresponding sub-container. The sub-container executes the compilation task according to the compilation parameters, obtains the offline neural network model and performs evaluation and analysis, determines the performance evaluation result, and returns the performance evaluation result and the offline neural network model to the main container.

[0061] The user's choices include the hardware platform, the original model, and the compilation parameters for the configuration.

[0062] Furthermore, Docker isolation is used to shield the differences in the compilation environment. The main container passes the necessary compilation parameters to each child Docker container, which then accepts and executes the compilation task. After compilation, preliminary performance analysis is performed, and the results are returned.

[0063] Specifically, it includes one main container and four child containers, arranged in a master-slave configuration. Before constructing the containers, port numbers are specified between them. Data is transmitted via sockets to handle the most basic compilation parameter passing.

[0064] Parameter passing can be implemented graphically by the main container. Users only need to add the necessary parameters and do not need to care about the compilation and execution principles. At the same time, when selecting to compile offline files for different platforms, corresponding sub-containers need to be connected, so the corresponding port numbers need to be associated. For example, if it is necessary to compile offline files for the Intel platform, after specifying the platform, the main container connects to the container where Intel is located, and the sub-container starts listening on the port, waiting for the parameters to be passed in.

[0065] To avoid data loss, files are transferred via shared folders. Within the main container, two shared folders are set up: the first, named "share," is used by the main container to share files or weight files that need to be compiled with child containers; the second shared folder, named "return," is used to return the compiled offline model, log files, and performance analysis results.

[0066] Within the sub-container, the shared folder also needs to be monitored. Different frameworks require monitoring different file formats for compilation; for example, the TensorFlow framework typically needs to monitor pb files. When the corresponding file is detected, the compilation command under the relevant framework is executed. After compilation, performance analysis is performed immediately, and the results are passed to the return folder for user use.

[0067] Furthermore, the sub-container executes compilation tasks based on compilation parameters to evaluate and analyze the offline neural network model, specifically including:

[0068] (1) The sub-container executes the compilation task according to the compilation parameters to generate an offline neural network model.

[0069] (2) Calculate the total execution time of the model inference by considering the memory access time, computation time, and overlap time during the offline neural network model runtime; the total execution time of the model inference is expressed as:

[0070] T exec =T math +T mem -T overlap

[0071] Among them, T exec T represents the total execution time of the model inference. mem T is the memory access time. math To calculate the elapsed time, T overlap For overlapping time.

[0072] Due to the parallel nature of the operation, memory access and computation can overlap in time. Therefore, this embodiment takes into account the total execution time of model inference during computation.

[0073] (3) Determine whether the total execution time of the model inference meets the preset production demand index; if yes, execute (4), otherwise execute (5).

[0074] (4) Return the offline neural network model, log files, and performance evaluation results to the main container.

[0075] (5) Calculate the arithmetic strength and operation byte ratio of the offline neural network model, and determine the constraint type of the offline neural network model based on the arithmetic strength and operation byte ratio; generate optimization suggestions based on the constraint type and return them to the main container so that the user can correct the model.

[0076] Arithmetic strength is expressed as:

[0077] T math =ops / BW mem

[0078] Where ops represents the computation time and number of operations, BW mem This refers to the processor's computing bandwidth.

[0079] The operand-byte ratio is expressed as:

[0080] T mem = bytes / BW mem

[0081] Where bytes is the memory bandwidth, BW mem This refers to the processor's computing bandwidth.

[0082] When the arithmetic strength is greater than the operand ratio, and the difference between the arithmetic strength and the operand ratio is greater than 50, the offline neural network model is constrained by computation. When the arithmetic strength is less than the operand ratio, and the difference between the arithmetic strength and the operand ratio is greater than 50, the offline neural network model is constrained by memory. If the AI ​​model's workload is insufficient and it cannot provide enough parallelism, the processor cannot be fully utilized, and the model's constrained type does not belong to either of the above two categories, but rather to latency-constrained.

[0083] If the constraint type is computationally constrained, optimization suggestions include model compression, model optimization, distributed computing, and hardware optimization; if the constraint type is memory constrained, optimization suggestions include batch processing, streaming data processing, data structure optimization, and replanning memory reclamation and management; if the constraint type is latency constrained, optimization suggestions include asynchronous computing, cloud-edge collaboration, and memory latency hiding.

[0084] (6) After adjusting the model's configuration parameters according to the optimization suggestions, the user executes (1)-(5).

[0085] For example, and in conjunction with a practical application, the specific process of this step will be further explained as follows:

[0086] S101. When a user starts a service in the main container, the main container can allocate the corresponding sub-containers based on the user's platform selection. After selecting the corresponding platform, the framework is selected. Different frameworks correspond to different parameter options for saving and passing to the sub-containers.

[0087] The main container is equipped with an interactive interface, which has three sub-functional modules: offline compilation, file management, and offline inference. The platform refers to edge hardware computing devices, such as Intel platforms, Atlas platforms, and Cambricon ML220 platforms. The framework is the software framework used to build neural networks, such as TensorFlow, ONNX, and Caffe.

[0088] S102. After configuring the parameters, create the project. When creating the project, generate the image, model, and return file frameworks for the project, and ask the user to select the neural network model file and parameter weight file to be compiled.

[0089] S103. When executing a user request and starting a task, the main container requests to connect to a specified sub-container or all sub-containers through the socket service. After the connection is successfully established, the parameters configured in step 1 are sent. At the same time, a file system service is established to load the files to be compiled into the specified shared folder for the sub-containers to receive the files for compilation.

[0090] S104. After the sub-container confirms the link is established, it begins to receive parameters. When it confirms that there is a file to be executed in the shared folder, it begins to read and identify the file information, determine the compilation form corresponding to the file, import the parameter information and configuration file, and begin to execute model quantization and compilation.

[0091] After the offline neural network model is successfully compiled, it is evaluated according to the defined performance analysis method, and the evaluation results are given according to the evaluation method.

[0092] After the evaluation is completed, the offline neural network model, compilation log, evaluation results, etc. are returned to the return shared folder for the main container to analyze. If the offline compilation fails, the log information is returned directly for the user to adjust the compilation parameters.

[0093] S2. Based on the performance evaluation results, select the optimal edge device. The main container deploys the offline neural network model to the optimal edge device. The optimal edge device runs the offline neural network model to perform inference tasks and returns the inference results to the main container.

[0094] For example, based on the final performance evaluation result (total execution time of model inference), combined with the computing power and resource overhead of the edge computing device, the user selects the optimal edge device that meets the requirements. The main container sends the inference application to the optimal edge device via RPC according to the user's selection. The edge side reads in the data through the data acquisition module and performs preprocessing.

[0095] Offline deployment is performed on the host side of the optimal edge device. The collected data is imported through the data acquisition module, and the edge module processes the data through the deployed offline neural network model, runs the inference task, and returns the inference results for post-processing or data sharing.

[0096] Inference applications refer to applications made using the offline neural network model. For example, an inference application uses the offline neural network model to detect car license plates and extract license plate information, returning the inference result, i.e., the license plate information, for subsequent operations.

[0097] Example 2

[0098] This embodiment discloses a cloud-based collaborative neural network model deployment system, based on the cloud-based collaborative neural network model deployment method described in Embodiment 1. It includes a server and an optimal edge device. The server, in response to user selection, loads the original model and configured compilation parameters into a main container and matches the corresponding sub-container. The sub-container executes a compilation task according to the compilation parameters, obtains an offline neural network model, performs evaluation and analysis, determines the performance evaluation result, and returns the performance evaluation result and the offline neural network model to the main container. The main container selects the optimal edge device based on the performance evaluation result and deploys the offline neural network model to the optimal edge device.

[0099] The optimal edge device is used to run offline neural network models to perform inference tasks and return the inference results to the main container.

[0100] Furthermore, the optimal edge device includes a data acquisition module, a host, an edge module, and a power supply module; the host and the edge module are connected via a serial bus and are both powered by the power supply module. The host imports the offline neural network model through the data acquisition module and deploys the offline neural network model to the edge module.

[0101] Users only need to operate the host and control the edge module through RPC. Data is read in by the image acquisition module, and the edge module performs corresponding processing according to the data stream. Direct control by the host avoids the differences caused by different hardware architectures.

[0102] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment, whereby a series of operational steps are performed to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0106] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for deploying a neural network model based on cloud-based collaboration, characterized in that, include: In response to the user's selection, the original model and the configured compilation parameters are loaded into the main container and matched with the corresponding sub-containers; The sub-container executes the compilation task according to the compilation parameters, obtains the offline neural network model and performs evaluation and analysis, determines the performance evaluation result, and returns the performance evaluation result and the offline neural network model to the main container; The evaluation and analysis of offline neural network models specifically includes: The total execution time of the model inference is calculated by measuring the memory access time, computation time, and overlap time during the offline neural network model runtime. Determine whether the total execution time of the model inference meets the preset production demand indicators; If not, calculate the arithmetic strength and operation-byte ratio of the offline neural network model. Based on the arithmetic strength and operation-byte ratio, determine the constraint type of the offline neural network model, including: When the arithmetic strength is greater than the operand ratio, and the difference between the arithmetic strength and the operand ratio is greater than a preset threshold, the offline neural network model is classified as computationally constrained. When the arithmetic strength is less than the operand ratio, and the difference between the arithmetic strength and the operand ratio is greater than a preset threshold, the offline neural network model is classified as memory-constrained. Otherwise, the constraint type of the offline neural network model is delay-constrained; Based on the type of constraint, optimization suggestions are generated to help users correct the model; The arithmetic strength and operand-to-byte ratio of the offline neural network model are specifically as follows: The arithmetic strength of the offline neural network model is obtained based on the computation time, number of operations, and processor bandwidth of the offline neural network model. Based on the processor's computing bandwidth and memory bandwidth, obtain the operation byte ratio of the offline neural network model; The process of generating optimization suggestions based on the constraint type includes: If the constraint type is computationally constrained, the optimization suggestions include model compression, model optimization, distributed computing, and hardware optimization. If the constraint type is memory-constrained, optimization suggestions include batch processing, streaming data processing, optimizing data structures, and replanning memory reclamation and management; If the constraint type is latency-constrained, optimization suggestions include asynchronous computing, cloud-edge collaboration, and memory latency hiding; The number of sub-containers is multiple, each corresponding to a different hardware platform; the main container selects the optimal edge device based on the performance evaluation results and deploys the offline neural network model to the optimal edge device; The optimal edge device runs an offline neural network model to perform inference tasks and returns the inference results to the main container.

2. The cloud-based collaborative neural network model deployment method as described in claim 1, characterized in that, The main container sends compilation parameters to the child container while simultaneously creating the first and second shared folders; The main container shares the original model to be compiled with the child container through the first shared folder, and the main container obtains the offline neural network model and performance analysis results returned by the child container through the second shared folder; The child container listens to the first and second shared folders.

3. The method for deploying a cloud-based collaborative neural network model as described in claim 1, characterized in that, If the performance evaluation results are unsatisfactory, the following steps are also taken before selecting the optimal edge device: In response to user selection, update the original model and / or configuration parameters to allow the sub-container to correct the offline neural network model until the performance evaluation results meet the requirements.

4. A cloud-based collaborative neural network model deployment system, characterized in that, Includes servers and optimal edge devices; The server responds to the user's selection by loading the original model and configured compilation parameters into the main container and matching the corresponding sub-container. The sub-container executes the compilation task according to the compilation parameters, obtains the offline neural network model, performs evaluation and analysis, determines the performance evaluation result, and returns the performance evaluation result and the offline neural network model to the main container. The evaluation and analysis of offline neural network models specifically includes: The total execution time of the model inference is calculated by measuring the memory access time, computation time, and overlap time during the offline neural network model runtime. Determine whether the total execution time of the model inference meets the preset production demand indicators; If not, calculate the arithmetic strength and operation-byte ratio of the offline neural network model. Based on the arithmetic strength and operation-byte ratio, determine the constraint type of the offline neural network model, including: When the arithmetic strength is greater than the operand ratio, and the difference between the arithmetic strength and the operand ratio is greater than a preset threshold, the offline neural network model is classified as computationally constrained. When the arithmetic strength is less than the operand ratio, and the difference between the arithmetic strength and the operand ratio is greater than a preset threshold, the offline neural network model is classified as memory-constrained. Otherwise, the constraint type of the offline neural network model is delay-constrained; Based on the type of constraint, optimization suggestions are generated to help users correct the model; The arithmetic strength and operand-to-byte ratio of the offline neural network model are specifically as follows: The arithmetic strength of the offline neural network model is obtained based on the computation time, number of operations, and processor bandwidth of the offline neural network model. Based on the processor's computing bandwidth and memory bandwidth, obtain the operation byte ratio of the offline neural network model; The process of generating optimization suggestions based on the constraint type includes: If the constraint type is computationally constrained, the optimization suggestions include model compression, model optimization, distributed computing, and hardware optimization. If the constraint type is memory-constrained, optimization suggestions include batch processing, streaming data processing, optimizing data structures, and replanning memory reclamation and management; If the constraint type is latency-constrained, optimization suggestions include asynchronous computing, cloud-edge collaboration, and memory latency hiding; The number of sub-containers is multiple, each corresponding to a different hardware platform; the main container selects the optimal edge device based on the performance evaluation results; and deploys the offline neural network model to the optimal edge device; The optimal edge device is used to run offline neural network models to perform inference tasks and return the inference results to the main container.

5. The cloud-based collaborative neural network model deployment system as described in claim 4, characterized in that, The optimal edge device includes a data acquisition module, a host, and an edge module; The offline neural network model is deployed to the edge module via the host. The data acquisition module collects data and transmits it to the edge module. The edge module runs the offline neural network model to process the data, performs inference tasks, and returns the inference results to the main container.

Images