Model deployment method and apparatus, electronic device, storage medium, and product
By limiting the range of activation function output values and format conversion, the problems of reduced accuracy and hardware mismatch caused by low-bit quantization are solved, and the image saliency segmentation model can be run efficiently and achieve the required accuracy on the terminal side.
Patent Information
- Application Number
- CN202411910598.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2026-06-23
AI Technical Summary
When deploying an image saliency segmentation model on the terminal side, low bit quantization leads to reduced model accuracy, which cannot meet the accuracy requirements, and the model format is incompatible with the hardware, resulting in low running efficiency.
By limiting the output value range of the preset activation function and converting the format of the image saliency segmentation model to a format supported by the target chip, including the compilation and deployment process, the model can be adapted to the target chip.
It improves the accuracy and inference efficiency of the image saliency segmentation model, meets the accuracy requirements, and runs efficiently on the target chip.
Smart Images
Figure CN122265306A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing technology, and in particular to a model deployment method, apparatus, electronic device, storage medium, and product. Background Technology
[0002] Image saliency segmentation models can identify and segment salient regions from images or videos. They have wide applications in fields such as photography, image matting, autonomous driving, and medical image analysis.
[0003] When deploying an image saliency segmentation model on the terminal side, low-bit quantization is usually performed to adapt to the hardware requirements of the terminal side. However, low-bit quantization will reduce the accuracy of the model, resulting in the image saliency segmentation model after low-bit quantization failing to meet the expected accuracy requirements. Summary of the Invention
[0004] To overcome the problems existing in related technologies, this disclosure provides a model deployment method, apparatus, electronic device, storage medium, and product.
[0005] According to a first aspect of the present disclosure, a model deployment method is provided, comprising:
[0006] Obtain an image saliency segmentation model; the image saliency segmentation model is used to perform saliency segmentation on the input image, the image saliency segmentation model includes a preset activation function, the output value of the preset activation function is within a preset range, and the maximum value of the preset range is a first preset value;
[0007] The image saliency segmentation model is converted into a target model format supported by the target chip.
[0008] The image saliency segmentation model in the target model format is deployed on the target chip.
[0009] In some embodiments, deploying the image saliency segmentation model in the target model format on the target chip includes:
[0010] The image saliency segmentation model in the target model format is compiled to obtain a target model file, which is used to characterize the image saliency segmentation model.
[0011] Based on the target model file, the image saliency segmentation model in the target model format is deployed on the target chip.
[0012] In some embodiments, compiling the image saliency segmentation model in the target model format to obtain the target model file includes:
[0013] The image saliency segmentation model is compiled to obtain the model library file corresponding to the image saliency segmentation model;
[0014] The target model file is generated based on the model library file and the hardware configuration parameters of the target chip.
[0015] In some embodiments, the method further includes:
[0016] The image saliency segmentation model is invoked to process the image to be processed, resulting in a saliency segmentation image corresponding to the image to be processed.
[0017] In some embodiments, the method further includes:
[0018] The image saliency segmentation model is processed by a target processing unit to accelerate inference; the target processing unit is a neural network processing unit supported by the target chip.
[0019] In some embodiments, the training process of the image saliency segmentation model includes:
[0020] Acquire training data, which includes sample images of multiple categories and sample saliency segmentation images corresponding to each sample image;
[0021] The sample image is input into the initial image saliency segmentation model to obtain the predicted saliency segmentation image corresponding to the sample image;
[0022] Based on the sample saliency segmentation image and the predicted saliency segmentation image, the parameters of the initial image saliency segmentation model are adjusted to obtain the image saliency segmentation model.
[0023] In some embodiments, the image saliency segmentation model includes a preset encoder and a preset decoder, and the step of inputting the sample image into the initial image saliency segmentation model to obtain the predicted saliency segmentation image corresponding to the sample image includes:
[0024] The preset encoder is used to extract features from the sample image to obtain the image feature information of the sample image;
[0025] The image feature information is parsed using the preset decoder to obtain the predicted saliency segmentation image.
[0026] In some embodiments, the preset encoder includes an inverted bottleneck block.
[0027] In some embodiments, when the input value of the preset activation function is greater than or equal to a first preset value, the output value of the preset activation function is the first preset value; when the input value of the preset activation function is less than or equal to a second preset value, the output value of the preset activation function is the second preset value; the first preset value is the maximum value of the preset output value range, and the second preset value is the minimum value of the preset output value range.
[0028] According to a second aspect of the present disclosure, a model deployment apparatus is provided, comprising:
[0029] The acquisition module is configured to acquire an image saliency segmentation model; the image saliency segmentation model is used to perform saliency segmentation on the input image, the image saliency segmentation model includes a preset activation function, the output value of the preset activation function is within a preset range, and the maximum value of the preset range is a first preset value;
[0030] The conversion module is configured to convert the model format of the image saliency segmentation model into a target model format supported by the target chip;
[0031] The processing module is configured to deploy the image saliency segmentation model in the target model format on the target chip.
[0032] According to a third aspect of the present disclosure, an electronic device is provided, comprising:
[0033] processor;
[0034] Memory used to store processor-executable instructions;
[0035] The processor is configured to execute the model deployment method as described in the first aspect of this disclosure.
[0036] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the model deployment method as described in the first aspect of the present disclosure.
[0037] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the model deployment method as described in the first aspect of the present disclosure.
[0038] The method described above, as disclosed in this invention, has the following beneficial effects:
[0039] The method provided in this disclosure can limit the size of the output value of the preset activation function, thereby avoiding the problem of decreased accuracy after model quantization caused by a large output value, ensuring the stability and accuracy of the output value, and improving the accuracy of the image saliency segmentation model to meet the accuracy requirements of the image saliency segmentation model. Furthermore, by converting the model format of the image saliency segmentation model to the target model format supported by the target chip, the image saliency segmentation model can run efficiently on the target chip, thereby improving the inference efficiency and inference performance of the image saliency segmentation model.
[0040] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0041] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0042] Figure 1 This is a flowchart illustrating a model deployment method according to an exemplary embodiment.
[0043] Figure 2 This is a schematic diagram of an activation function according to an exemplary embodiment.
[0044] Figure 3 This is a flowchart illustrating a model deployment method according to an exemplary embodiment.
[0045] Figure 4 This is a schematic diagram illustrating a model deployment method according to an exemplary embodiment.
[0046] Figure 5 This is a flowchart illustrating a model deployment method according to an exemplary embodiment.
[0047] Figure 6 This is a schematic diagram illustrating the use of an image saliency segmentation model according to an exemplary embodiment.
[0048] Figure 7 This is a flowchart illustrating a model deployment method according to an exemplary embodiment.
[0049] Figure 8 This is a block diagram of a model device illustrated according to an exemplary embodiment.
[0050] Figure 9 This is a block diagram of an electronic device according to an exemplary embodiment. Detailed Implementation
[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0052] Image saliency segmentation models can identify and segment salient regions from images or videos. They have wide applications in fields such as photography, image matting, autonomous driving, and medical image analysis.
[0053] When deploying image saliency segmentation models on the terminal side, low-bit quantization is typically performed to adapt to the hardware requirements. Low-bit quantization involves converting model parameters from high-precision to low-precision parameters, thereby reducing the model's parameter size, memory consumption, and inference latency. However, low-precision parameters cannot accurately describe a large numerical range, leading to a decrease in the accuracy of the low-bit quantized model. Image saliency segmentation models require pixel-level precision inference results, which low-bit quantization cannot meet. Furthermore, during model deployment, if the model format differs from the terminal's hardware format, the image saliency segmentation model cannot run efficiently on the terminal, resulting in low operational efficiency.
[0054] To address the aforementioned issues, this disclosure provides a model deployment method. This method can acquire an image saliency segmentation model. The image saliency segmentation model is used to perform saliency segmentation on an input image. The image saliency segmentation model includes a preset activation function, the output value of which is within a preset range, and the maximum value within the preset range is a first preset value. The method converts the model format of the image saliency segmentation model to a target model format supported by the target chip. The image saliency segmentation model in the target model format is then deployed on the target chip. The method provided by this disclosure can limit the size of the output value of the preset activation function, thereby avoiding the problem of decreased accuracy after model quantization due to a large output value, ensuring the stability and accuracy of the output value, and thus improving the accuracy of the image saliency segmentation model to meet its accuracy requirements. Furthermore, by converting the model format of the image saliency segmentation model to the target model format, the image saliency segmentation model can run efficiently on the target chip, thereby improving the inference efficiency and performance of the image saliency segmentation model.
[0055] The model deployment method provided in this disclosure is executed by an electronic device, which may specifically be a mobile phone, tablet computer, laptop, smart robot, smart wearable device, or other smart device. In addition, the electronic device is equipped with various hardware resources and an energy storage device that provides power for the operation of these hardware resources.
[0056] Figure 1 This is a flowchart illustrating a model deployment method according to an exemplary embodiment, performed by an electronic device, see [link to flowchart]. Figure 1 The method includes the following steps:
[0057] Step S101: Obtain the image saliency segmentation model; the image saliency segmentation model is used to perform saliency segmentation on the input image. The image saliency segmentation model includes a preset activation function. The output value of the preset activation function is within a preset range. The maximum value of the preset range is a first preset value.
[0058] Image saliency segmentation models can be used to identify and segment salient regions in an image. These models can effectively identify salient regions in a scene, thus solving complex visual problems and providing users with scene understanding.
[0059] It should be noted that the image saliency segmentation model is a trained model. The specific training process can be found in steps S501-S503 below, and will not be repeated here.
[0060] Model quantization is required during model deployment. Model quantization involves converting floating-point parameters (such as weights and activation values) in the model into lower-precision parameters, for example, from 32-bit floating-point numbers (FP32) to 16-bit floating-point numbers (FP16) or 8-bit integers (INT8). Model quantization makes the model more lightweight, thereby reducing its storage size and accelerating inference speed.
[0061] Existing activation functions do not limit the range of output values. Taking the ReLU activation function as an example, when the input value is greater than 0, the output is that input value; conversely, when the input value is less than or equal to 0, the output is 0. That is, the output value range of the ReLU activation function is [0, +∞). Since low-precision parameters cannot accurately describe a large numerical range, the precision of the output value of the activation function after model quantization will decrease, thus affecting the segmentation performance of the image saliency segmentation model.
[0062] In some embodiments, this embodiment can limit the range of the output value of the activation function, thereby limiting the size of the output value of the preset activation function, thus avoiding the problem of decreased accuracy after quantization due to a large output value, and ensuring the stability and accuracy of the output value.
[0063] In some embodiments, the output value of the preset activation function can be limited to a preset range. The preset range can be set and selected based on actual needs. For example, the first preset value can be 6, 7, or 8. In one example, when the input value of the preset activation function is greater than or equal to the first preset value, the output value of the preset activation function is the first preset value; when the input value of the preset activation function is less than or equal to the second preset value, the output value of the preset activation function is the second preset value. The first preset value is the maximum value of the preset output value range, and the second preset value is the minimum value of the preset output value range. The second preset value can be 0. For example, when the first preset value is 6 and the second preset value is 0, when the input value is less than or equal to 0, the output value of the preset activation function is 0, and when the output value is greater than or equal to 6, the output value of the preset activation function is 6.
[0064] Taking the first preset value of 6 as an example, the following is given: Figure 2 The diagram shows the ReLU activation function and the ReLU6 activation function (with a maximum value of 6 in the preset range). Figure 2 In this diagram, the x-axis refers to the input value, and the y-axis refers to the output value. Figure 2 It can be determined that for the ReLU activation function, when x is less than or equal to 0, the corresponding y value of the ReLU activation function is 0; when x is greater than 0 and x is less than 6, the corresponding y value of the ReLU activation function is the same as the x value; when x is greater than or equal to 6, the corresponding y value of the ReLU activation function is the same as the x value.
[0065] For the ReLU6 activation function, when x is less than or equal to 0, the corresponding y value of the ReLU activation function is 0; when x is greater than 0 and x is less than 6, the corresponding y value of the ReLU activation function is the same as the x value; when x is greater than or equal to 6, the corresponding y value of the ReLU activation function is 6.
[0066] Therefore, compared to the ReLU activation function, the ReLU6 activation function can limit the size of the activation function's output value, thereby reducing the accuracy loss caused by model quantization, improving the model's computational accuracy, and thus meeting the accuracy requirements of image saliency segmentation models.
[0067] Step S102: Convert the model format of the image saliency segmentation model to the target model format supported by the target chip.
[0068] When the format of the image saliency segmentation model differs from that of the target model, the image saliency segmentation model is not compatible with the target chip. In this case, if the image saliency segmentation model is deployed on the target chip, some functions of the image saliency segmentation model will be unusable or perform poorly on the target chip due to the incompatibility, thus failing to achieve optimal inference performance.
[0069] Therefore, in order to ensure that the model can be executed correctly on the target chip, during the deployment of the image saliency segmentation model, the model format of the image saliency segmentation model can be converted based on the target model format supported by the target chip, so as to make the image saliency segmentation model more suitable for the target chip.
[0070] In some embodiments, the model format of an image saliency segmentation model can be determined based on the model framework of the image saliency segmentation model. Here, the model framework refers to a collection of tools and libraries for building, training, evaluating, and deploying machine learning or deep learning models.
[0071] Specifically, the file format of the image saliency segmentation model can be determined based on its framework, and the model format of the image saliency segmentation model can be determined based on the file format. Optionally, the framework of the image saliency segmentation model can be a PyTorch framework (a deep learning database), and the default file format of the PyTorch framework is .pt or .pth format. Correspondingly, when the framework of the image saliency segmentation model can be a PyTorch framework, the model format of the image saliency segmentation model can be .pt or .pth format.
[0072] In some embodiments, the target chip can be any type of electronic device chip, and different electronic chips support different model formats. For example, the target chip supports the Qualcomm Neural Network (QNN) framework, and the model format is QNN. The QNN framework supports model conversion from the PyTorch framework to the Qualcomm computing platform and performs low-bit quantization of the model, so that the PyTorch framework image saliency segmentation model can run on the target chip.
[0073] Taking the QNN framework supported by the target chip as an example, the model conversion process is explained in detail below:
[0074] The image saliency segmentation model was converted from pth format to Open Neural Network Exchange (ONNX) format using the torch.onnx library provided by the PyTorch framework, resulting in the converted ONNX format image saliency segmentation model. Then, the qnn-onnx-converter tool (command-line tool) was used to convert the ONNX format image saliency segmentation model to the QNN format image saliency segmentation model, and model quantization was performed to obtain the converted QNN format image saliency segmentation model.
[0075] Step S103: Deploy the image saliency segmentation model in the target model format on the target chip.
[0076] Different electronic chip architectures have different computational and storage requirements for models. When deploying an image saliency segmentation model on a target chip, the image saliency segmentation model needs to be compiled. Compilation can perform operations such as operator fusion and memory optimization on the model, thereby reducing computational resource consumption and improving the model's running efficiency.
[0077] The method provided in this disclosure can limit the size of the output value of the preset activation function, thereby avoiding the problem of decreased accuracy after model quantization caused by a large output value, ensuring the stability and accuracy of the output value, and improving the accuracy of the image saliency segmentation model to meet the accuracy requirements of the image saliency segmentation model. Furthermore, by converting the model format of the image saliency segmentation model to the target model format, the image saliency segmentation model can run efficiently on the target chip, thereby improving the inference efficiency and inference performance of the image saliency segmentation model.
[0078] In some embodiments, when deploying an image saliency segmentation model in target pattern format on a target chip, the image saliency segmentation model needs to be compiled. The following is an example... Figure 3 The illustrated embodiment describes the deployment process of an image saliency segmentation model in the target model format.
[0079] Figure 3 This is a flowchart illustrating a model deployment method according to an exemplary embodiment, performed by an electronic device, see [link to flowchart]. Figure 3 The method includes the following steps.
[0080] Step S301: Compile the image saliency segmentation model in the target model format to obtain the target model file, which is used to characterize the image saliency segmentation model.
[0081] In some embodiments, compiling an image saliency segmentation model in a target model format includes: compiling the image saliency segmentation model to obtain a model library file corresponding to the image saliency segmentation model; and generating a target model file based on the model library file and the hardware configuration parameters of the target chip.
[0082] A model library file is a file generated after compiling a model. It includes the compilation results of the model and enables resource sharing, thereby improving design and development efficiency. Optionally, a model library file may include: the model's source code; model parameters, such as those obtained during model training; execution configuration information, which indicates how the model runs in a specific hardware and software environment; source code and intermediate representation; and metadata, such as the model's name and version.
[0083] In some embodiments, the image saliency segmentation model can be compiled using the qnn-model-lib-generator tool (a command-line tool) to obtain a model library file.
[0084] The hardware configuration parameters of a target chip can refer to the parameters and components involved in the chip design and manufacturing process, such as the type, number, and operating mode of the target chip's processor; the processor's operating speed; the cache configuration; the input / output interfaces; and the processor's data processing capabilities, such as data transfer rate.
[0085] The target model file refers to the model file corresponding to the image saliency segmentation model. It should be noted that the target model file is a context binary file specific to the target chip, storing data such as the execution environment, operation information involved in the image saliency segmentation model, and the order of operations involved in the image saliency segmentation model. The target model file also contains pre-compiled model information and optimizations.
[0086] In some embodiments, the qnn-context-binary-generator tool (a command-line tool) can be used to optimize and compile the model library file to obtain the target model file. Specifically, the model library file, the hardware configuration and parameters of the target chip, and the name of the output binary file can be used as parameters for the qnn-context-binary-generator tool. Based on the above parameters, the qnn-context-binary-generator tool can generate a context binary file to obtain the target model file.
[0087] Step S302: Based on the target model file, deploy the image saliency segmentation model in the target model format on the target chip.
[0088] In some embodiments, the target model file is a model file relative to the target chip. Accordingly, the target model file can be deployed on the target chip to realize the deployment of the image saliency segmentation model in the target model format on the target chip.
[0089] In some embodiments, after deployment, an image saliency segmentation model can be invoked to process the image to be processed, resulting in a saliency segmentation image corresponding to the image to be processed. In one example, the image saliency segmentation model can be invoked in the target application through the QNN interface to process the image to be processed.
[0090] In some embodiments, a target processing unit can be used to accelerate the inference process of the image saliency segmentation model. The target processing unit is a neural network processing unit supported by the target chip, thereby accelerating the inference of the image saliency segmentation model and improving its inference efficiency. In one example, the target processing unit can be an NPU (Neural-network Processing Unit), and the configuration file used by the target processing unit to accelerate the processing of the image saliency segmentation model can be libQnnHtp.so.
[0091] The method provided in this disclosure can optimize the compilation of the model library file based on the model library file and the hardware configuration parameters of the target chip. Thus, when the image saliency segmentation model is called on the target chip, there is no need to repeatedly compile and optimize the model, thereby accelerating the execution speed of the image saliency segmentation model. Furthermore, the inference process of the image saliency segmentation model can be further accelerated through the target processing unit, thereby achieving inference acceleration.
[0092] Figure 4 This is a schematic diagram illustrating a model deployment method according to an exemplary embodiment. First, model conversion is performed, including: converting the PyTorch model architecture image saliency segmentation model to an ON NX model format image saliency segmentation model using the torch.onnx file; then, converting the ONNX model format image saliency segmentation model to a QNN model format image saliency segmentation model using the qnn-onnx-converter command-line tool. Next, model compilation is performed. The QNN model format image saliency segmentation model is compiled using the qnn-model-lib-generator command-line tool to obtain the corresponding model library file. Then, offline model caching is performed. Based on the model library file and the target chip's hardware configuration parameters, the target model file is generated using the qnn-context-binary-generator command-line tool. Finally, model invocation is performed. The image saliency segmentation model represented by the target model file can be invoked through the target interface to process the image to be processed and obtain the processing result. The target interface can be the interface corresponding to the target chip.
[0093] In some embodiments, the following is by Figure 5 The illustrated example demonstrates the training process of an image saliency segmentation model.
[0094] Figure 5 This is a flowchart illustrating a model deployment method according to an exemplary embodiment, performed by an electronic device, see [link to flowchart]. Figure 5 The method includes the following steps:
[0095] Step S501: Obtain training data, which includes sample images of multiple categories and sample saliency segmentation images corresponding to each sample image.
[0096] In some embodiments, training sets for multiple categories can be obtained and mixed to obtain training data. The training set for each category includes sample images of the same category and sample saliency segmentation images corresponding to the sample images. The category of a sample image can be determined based on the content included in the sample image. For example, sample images containing human figures can be classified into the same category, sample images containing animals can be classified into the same category, and sample images containing buildings can be classified into the same category.
[0097] It should be noted that mixing training sets from multiple categories to obtain training data can increase the quantity and variety of training data, thereby improving the recognition and generalization capabilities of the trained image saliency segmentation model.
[0098] Step S502: Input the sample image into the initial image saliency segmentation model to obtain the predicted saliency segmentation image corresponding to the sample image.
[0099] The initial image saliency segmentation model can use CTDNet (a network architecture for salient object detection) as the baseline model. The image saliency segmentation model includes a pre-defined encoder and a pre-defined decoder. The encoder performs feature extraction, for example, using a convolutional neural network. The decoder incorporates complementary trilateral decoding branches, analyzes the image feature information extracted by the encoder, performs fusion at different levels, outputs a salient object edge map, and fuses the outputs from different levels to obtain the predicted saliency segmentation image.
[0100] The preset encoder can be a MobileNet v4 encoder. The MobileNet v4 encoder includes an inverted bottleneck block, which incorporates multiple architectural designs, providing spatial and channel flexibility, thus ensuring both accuracy and speed in feature extraction. Therefore, the MobileNet v4 encoder can improve the computational efficiency of image saliency segmentation models. Furthermore, this embodiment does not limit the type of preset decoder, and will not be elaborated further here.
[0101] In some embodiments, a preset encoder can be used to extract features from the sample image to obtain image feature information of the sample image, and a preset decoder can be used to parse the image feature information to obtain a predicted saliency segmentation image.
[0102] Step S503: Based on the sample saliency segmentation image and the predicted saliency segmentation image, adjust the parameters of the initial image saliency segmentation model to obtain the image saliency segmentation model.
[0103] In some embodiments, the parameters of a preset encoder and a preset decoder can be adjusted based on the sample saliency segmentation image and the predicted saliency segmentation image to obtain an image saliency segmentation model with adjusted parameters.
[0104] In some embodiments, after obtaining the image saliency segmentation model, test data can be used to test the image saliency segmentation model. For example, test data is input into the image saliency segmentation model to obtain a test saliency segmentation image. The accuracy of the image saliency segmentation model is determined by comparing the test saliency segmentation image and the sample saliency segmentation image. If the accuracy of the image saliency segmentation model is greater than a preset threshold, the image saliency segmentation model is applied to subsequent processes; if the accuracy of the image saliency segmentation model is less than or equal to the preset threshold, the parameters of the image saliency segmentation model are continuously adjusted until the accuracy of the image saliency segmentation model is greater than or equal to the preset threshold.
[0105] Figure 6 A schematic diagram illustrating the use of an image saliency segmentation model is shown. When using the image saliency segmentation model, the image to be processed can be input into the preset encoder of the image saliency segmentation model. The preset encoder can extract the image feature information of the image and send the image feature information to the preset decoder. The preset decoder can analyze the image feature information extracted by the preset encoder and output the saliency segmentation image corresponding to the image to be processed.
[0106] The method provided in this disclosure, the MobileNet v4 encoder, can improve the feature extraction speed of images, thereby improving the inference speed of the image saliency segmentation model.
[0107] Figure 7 This is a flowchart illustrating a model deployment method according to an exemplary embodiment, performed by an electronic device, see [link to flowchart]. Figure 7 The method includes the following steps:
[0108] Step S701: Obtain the image saliency segmentation model.
[0109] Step S702: Convert the model format of the image saliency segmentation model to the target model format supported by the target chip.
[0110] Step S703: Compile the image saliency segmentation model to obtain the model library file corresponding to the image saliency segmentation model.
[0111] Step S704: Generate the target model file based on the model library file and the hardware configuration parameters of the target chip.
[0112] Step S705: Based on the target model file, deploy the image saliency segmentation model in the target model format on the target chip.
[0113] Step S706: Call the image saliency segmentation model to process the image to be processed, and obtain the saliency segmentation image corresponding to the image to be processed.
[0114] The image saliency segmentation model obtained in step S701 is a pre-trained model. The training process of this model can be referred to in steps S501-S503 above, and will not be repeated here.
[0115] Figure 8 This is a block diagram illustrating a model deployment apparatus according to an exemplary embodiment, configured on an electronic device, see [link to relevant documentation]. Figure 8 The device includes:
[0116] The acquisition module 801 is configured to acquire an image saliency segmentation model; the image saliency segmentation model is used to perform saliency segmentation on the input image, the image saliency segmentation model includes a preset activation function, the output value of the preset activation function is within a preset range, and the maximum value of the preset range is a first preset value;
[0117] The conversion module 802 is configured to convert the model format of the image saliency segmentation model into the target model format supported by the target chip;
[0118] Processing module 803 is configured to deploy an image saliency segmentation model in target model format on the target chip.
[0119] In some embodiments, the processing module 803 is configured to:
[0120] The image saliency segmentation model in target model format is compiled to obtain the target model file, which is used to characterize the image saliency segmentation model.
[0121] Based on the target model file, the image saliency segmentation model in the target model format is deployed on the target chip.
[0122] In some embodiments, the processing module 803 is configured to:
[0123] The image saliency segmentation model is compiled to obtain the corresponding model library file;
[0124] The target model file is generated based on the model library file and the hardware configuration parameters of the target chip.
[0125] In some embodiments, the processing module 803 is further configured to:
[0126] The image saliency segmentation model is invoked to process the image to be processed, resulting in a saliency segmentation image corresponding to the image to be processed.
[0127] In some embodiments, the processing module 803 is further configured to:
[0128] The inference process of the image saliency segmentation model is accelerated by using a target processing unit; the target processing unit is a neural network processing unit supported by the target chip.
[0129] In some embodiments, the processing module 803 is further configured to:
[0130] Acquire training data, which includes sample images of multiple categories and sample saliency segmentation images corresponding to each sample image;
[0131] The sample image is input into the initial image saliency segmentation model to obtain the predicted saliency segmentation image corresponding to the sample image;
[0132] Based on the sample saliency segmentation image and the predicted saliency segmentation image, the parameters of the initial image saliency segmentation model are adjusted to obtain the image saliency segmentation model.
[0133] In some embodiments, the image saliency segmentation model includes a preset encoder and a preset decoder, and the processing module 803 is further configured to:
[0134] The features of the sample image are extracted using a preset encoder to obtain the image feature information of the sample image;
[0135] The image feature information is parsed using a pre-defined decoder to obtain a predicted saliency segmentation image.
[0136] In some embodiments, the preset encoder includes an inverted bottleneck block.
[0137] In some embodiments, when the input value of the preset activation function is greater than or equal to a first preset value, the output value of the preset activation function is the first preset value; when the input value of the preset activation function is less than or equal to a second preset value, the output value of the preset activation function is the second preset value; the first preset value is the maximum value of the preset output value range, and the second preset value is the minimum value of the preset output value range.
[0138] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0139] This disclosure also provides an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the model deployment method described in the above embodiments.
[0140] Figure 9 This is a block diagram of an electronic device 900 according to an exemplary embodiment.
[0141] Reference Figure 9 The electronic device 900 may include one or more of the following components: a processing component 902, a memory 904, a power supply component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.
[0142] Processing component 902 typically controls the overall operation of electronic device 900, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.
[0143] Memory 904 is configured to store various types of data to support the operation of electronic device 900. Examples of this data include instructions for any application or method operating on electronic device 900, contact data, phonebook data, messages, pictures, videos, etc. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0144] Power supply component 906 provides power to various components of electronic device 900. Power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 900.
[0145] Multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0146] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when electronic device 900 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.
[0147] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.
[0148] Sensor assembly 914 includes one or more sensors for providing state assessments of various aspects of electronic device 900. For example, sensor assembly 914 can detect the on / off state of electronic device 900, the relative positioning of components such as the display and keypad of electronic device 900, changes in position of electronic device 900 or a component of electronic device 900, the presence or absence of user contact with electronic device 900, orientation or acceleration / deceleration of electronic device 900, and temperature changes of electronic device 900. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.
[0149] Communication component 916 is configured to facilitate wired or wireless communication between electronic device 900 and other devices. Electronic device 900 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0150] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.
[0151] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, which can be executed by a processor 920 of an electronic device 900 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0152] A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the model deployment method provided by an exemplary embodiment of the present disclosure.
[0153] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the model deployment method described in the above embodiments.
[0154] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.
[0155] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A model deployment method, characterized in that, include: Obtain an image saliency segmentation model; the image saliency segmentation model is used to perform saliency segmentation on the input image, the image saliency segmentation model includes a preset activation function, the output value of the preset activation function is within a preset range, and the maximum value of the preset range is a first preset value; The image saliency segmentation model is converted into a target model format supported by the target chip. The image saliency segmentation model in the target model format is deployed on the target chip.
2. The deployment method of the model according to claim 1, characterized in that, The step of deploying the image saliency segmentation model in the target model format on the target chip includes: The image saliency segmentation model in the target model format is compiled to obtain a target model file, which is used to characterize the image saliency segmentation model. Based on the target model file, the image saliency segmentation model in the target model format is deployed on the target chip.
3. The model deployment method according to claim 2, characterized in that, The process of compiling the image saliency segmentation model in the target model format to obtain the target model file includes: The image saliency segmentation model is compiled to obtain the model library file corresponding to the image saliency segmentation model; The target model file is generated based on the model library file and the hardware configuration parameters of the target chip.
4. The model deployment method according to claim 3, characterized in that, The method further includes: The image saliency segmentation model is invoked to process the image to be processed, resulting in a saliency segmentation image corresponding to the image to be processed.
5. The model deployment method according to claim 4, characterized in that, The method further includes: The image saliency segmentation model is processed by a target processing unit to accelerate inference; the target processing unit is a neural network processing unit supported by the target chip.
6. The model deployment method according to claim 1, characterized in that, The training process of the image saliency segmentation model includes: Acquire training data, which includes sample images of multiple categories and sample saliency segmentation images corresponding to each sample image; The sample image is input into the initial image saliency segmentation model to obtain the predicted saliency segmentation image corresponding to the sample image; Based on the sample saliency segmentation image and the predicted saliency segmentation image, the parameters of the initial image saliency segmentation model are adjusted to obtain the image saliency segmentation model.
7. The model deployment method according to claim 6, characterized in that, The image saliency segmentation model includes a preset encoder and a preset decoder. The step of inputting the sample image into the initial image saliency segmentation model to obtain the predicted saliency segmentation image corresponding to the sample image includes: The preset encoder is used to extract features from the sample image to obtain the image feature information of the sample image; The image feature information is parsed using the preset decoder to obtain the predicted saliency segmentation image.
8. The model deployment method according to claim 7, characterized in that, The preset encoder includes an inverted bottleneck block.
9. The model deployment method according to claim 1, characterized in that, When the input value of the preset activation function is greater than or equal to the first preset value, the output value of the preset activation function is the first preset value; when the input value of the preset activation function is less than or equal to the second preset value, the output value of the preset activation function is the second preset value; the first preset value is the maximum value of the preset output value range, and the second preset value is the minimum value of the preset output value range.
10. A model deployment device, characterized in that, include: The acquisition module is configured to acquire an image saliency segmentation model; the image saliency segmentation model is used to perform saliency segmentation on the input image, the image saliency segmentation model includes a preset activation function, the output value of the preset activation function is within a preset range, and the maximum value of the preset range is a first preset value; The conversion module is configured to convert the model format of the image saliency segmentation model into a target model format supported by the target chip; The processing module is configured to deploy the image saliency segmentation model in the target model format on the target chip.
11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the model deployment method as described in any one of claims 1-9.
12. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the model deployment method as described in any one of claims 1-9.
13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the model deployment method as described in any one of claims 1-9.