Lightweight method and device of U-net model

Abstract

The application relates to the technical field of deep learning, in particular to a lightweight method and device of a U-net model. The lightweight method comprises the following steps: acquiring a U-net model; adding a pre-pruning factor layer in the U-net model to obtain a first model; training the first model by using first sample data to obtain a second model; and pruning channels with parameters less than a threshold in the pre-pruning factor layer of the second model to obtain a third model. According to the embodiment of the application, the U-net model is pruned, the network model inference is accelerated, and the accuracy of the network model is ensured.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a lightweight method and device for a U-net model. Background Art

[0002] With the development of deep learning, neural network models are being applied to an increasing number of application scenarios. However, large neural network models are computationally intensive and require a lot of computing resources. This is particularly time-consuming on edge mobile devices, where computational resources are low.

[0003] Taking the portrait cutout application scenario as an example, in this application scenario, a neural network model is often used as an image feature extraction module. The neural network model has a better representation of the image and can achieve a good cutout effect. However, the neural network model has a large amount of computation and requires more computing resources. In edge mobile devices with low computational capacity, it is time-consuming and difficult to achieve real-time portrait cutout. Summary of the Invention

[0004] In view of this, an embodiment of the present application provides a lightweight method and device for a U-net model, which can solve at least one technical problem in the related art.

[0005] In a first aspect, an embodiment of the present application provides a lightweight method for a U-net model, including: obtaining a U-net model; adding a pre-clipping factor layer to the U-net model to obtain a first model; training the first model using first sample data to obtain a second model; and pruning channels in the second model whose parameters of the pre-clipping factor layer are less than a threshold to obtain a third model.

[0006] This embodiment provides a lightweight method for a U-net model, which adds a pre-pruning factor layer to the U-net model and performs pruning based on the trained parameters of the pre-pruning factor layer, thereby achieving lightweighting of the network model and ensuring a high accuracy while accelerating network model inference.

[0007] In some embodiments, adding a pre-clipping factor layer to the U-net model to obtain a first model includes: adding a pre-clipping factor layer after the first convolution unit of each convolution module in the encoder of the U-net model to obtain the first model.

[0008] In some embodiments, during the back propagation process of training the first model using the first sample data to obtain the second model, the parameters of each layer belonging to the U-net model in the first model are updated using the gradient of the loss function, and the parameters of the pre-clipping factor layer in the first model are updated using the gradient of the loss function and the gradient of the penalty term.

[0009] In some embodiments, pruning the channels whose parameters of the pre-clipping factor layer in the second model are less than a threshold value to obtain a third model includes: step S1, pruning the channels whose parameters of the pre-clipping factor layer in the second model are less than a threshold value to obtain a third model; step S2, if the third model does not meet the preset conditions, updating the threshold value and returning to execute the above step S1; step S3, if the third model meets the preset conditions, storing the third model.

[0010] In some embodiments, after pruning the channels whose parameters of the pre-clipping factor layer in the second model are less than a threshold to obtain the third model, the method further includes: fusing the pre-clipping factor layer retained in the third model with the convolution unit in front of it to obtain a fourth model.

[0011] In some embodiments, it also includes: using second sample data to tune and train the third model to obtain a fourth model, the amount of the second sample data is less than the amount of the first sample data, and the second sample data is sample data associated with the application scenario.

[0012] In some embodiments, the first sample data is labeled portrait cutout data, and the third model is a model for portrait cutout.

[0013] In the second aspect, an embodiment of the present application provides a lightweight device for a U-net model, including: an acquisition module for acquiring a U-net model; an addition module for adding a pre-clipping factor layer to the U-net model to obtain a first model; a training module for training the first model using first sample data to obtain a second model; and a pruning module for pruning channels in the second model whose parameters of the pre-clipping factor layer are less than a threshold to obtain a third model.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the lightweight method of the U-net model as described in any embodiment of the first aspect is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the lightweight method of the U-net model as described in any embodiment of the first aspect.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes the lightweight method of the U-net model as described in any embodiment of the first aspect.

[0017] It should be understood that the beneficial effects of the second to fifth aspects can be found in the relevant description of the embodiment of the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 This is a schematic diagram of a lightweight implementation method of a U-net model provided in one embodiment of the present application;

[0020] Figure 2 This is a structural diagram of a convolution module provided in one embodiment of the present application;

[0021] Figure 3 This is a structural diagram of a convolution module before and after adding a pre-clipping factor layer according to an embodiment of the present application;

[0022] Figure 4 This is a schematic diagram of a lightweight implementation method of a U-net model provided in another embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of a lightweight implementation method of a U-net model provided in another embodiment of the present application;

[0024] Figure 6 This is a schematic diagram of a lightweight implementation method of a U-net model provided in another embodiment of the present application;

[0025] Figure 7 This is a schematic structural diagram of a lightweight device of a U-net model provided in one embodiment of the present application;

[0026] Figure 8 This is a schematic structural diagram of a lightweight device of a U-net model provided in one embodiment of the present application;

[0027] Figure 9 This is a schematic structural diagram of a lightweight device of a U-net model provided in one embodiment of the present application;

[0028] Figure 10 This is a schematic structural diagram of a lightweight device of a U-net model provided in one embodiment of the present application;

[0029] Figure 11This is a schematic structural diagram of a lightweight device of a U-net model provided in one embodiment of the present application;

[0030] Figure 12 This is a schematic structural diagram of a lightweight device of a U-net model provided in one embodiment of the present application;

[0031] Figure 13 This is a structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0033] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of the present application include a particular feature, structure, or characteristic described in conjunction with that embodiment. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0035] In addition, in the description of this application, "a plurality of" means two or more. The terms "first" and "second" are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0036] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0037] At present, neural network models have a large amount of computation and require a lot of computing resources. In edge mobile devices with low computational capacity, it is very time-consuming and difficult to achieve real-time implementation.

[0038] In order to solve the technical problems of related technologies, the embodiments of the present application provide a lightweight method and device for a U-net model, which prunes the U-net model, thereby achieving lightweighting of the network model, and ensuring a high accuracy while accelerating the network model reasoning.

[0039] Figure 1 This is a schematic diagram of the implementation process of a lightweight method of a U-net model provided by an embodiment of the present application. The lightweight method of the U-net model in this embodiment can be executed by an electronic device. The electronic device includes but is not limited to computers, tablets, servers, mobile phones, cameras or wearable devices, etc. Among them, the server includes but is not limited to an independent server or a cloud server, etc. Figure 1 As shown, the lightweight method of the U-net model may include steps S110 to S140.

[0040] S110, obtain a U-net model.

[0041] The embodiment of the present application takes the U-net model as the object and prunes it to obtain a lightweight network model. The U-net model consists of an encoder on the left and a decoder on the right.

[0042] In one embodiment, the encoder uses a typical convolutional neural network structure, which repeatedly uses convolution modules. Figure 2 As shown in the figure, each convolutional module consists of two convolutional units C and one maximum pooling layer. Each pooling operation doubles the dimension of the feature map. The decoder first performs a deconvolution operation to halve the dimension of the feature map. It then concatenates the feature maps obtained by cropping the corresponding compression channels to form a feature map of twice the size. It then uses two convolutional layers for feature extraction, and repeats this structure. In the final output layer, two convolutional layers are used to map the 64-dimensional feature map to a 2D output map.

[0043] S120: Add a pre-clipping factor layer to the U-net model to obtain a first model.

[0044] The first model is based on the U-net model, with an additional pre-clipping factor layer connected after the convolutional layer. The pre-clipping factor layer is a convolutional layer with a kernel size of 1×1 and the same number of input and output channels as the convolutional layer it is connected to.

[0045] In one embodiment, a pre-clipping factor layer is added after the first convolution unit of each convolution module in the encoder of the U-net model to obtain the first model. Figure 3 As shown in Figure (a), the convolution module includes two convolution units with a convolution kernel of 3×3 size. A 1×1 pre-cropping factor layer is added after the first convolution unit in the convolution module, as shown in Figure 5. Figure 3 As shown in Figure (b).

[0046] It should be noted that Figure 3 Only an example of a convolution module including one channel is given. In actual applications, a convolution module usually includes multiple channels, such as 64, 128, 256 channels, etc., and a pre-clipping factor layer is added after the first convolution unit corresponding to each channel.

[0047] S130: Use the first sample data to train the first model to obtain a second model.

[0048] In some embodiments, before training the first model, the step of initializing the first model, i.e., initializing the parameters of the first model, may be further included. In one embodiment, the parameters of the original U-net model in the first model may be initialized. This application does not limit the specific values ​​of the parameters, but the parameters of the pre-clipping factor layer in the first model are initialized to the identity matrix to ensure that the output of the initialized first model is consistent with the original U-net model.

[0049] In the back propagation process of training the first model using the first sample data to obtain the second model, the parameters of each layer belonging to the U-net model in the first model are updated using the gradient of the loss function, and the parameters of the pre-clipping factor layer in the first model are updated using the gradient of the loss function and the gradient of the penalty term.

[0050] In one embodiment, when training the first model, supervised learning is performed using labeled portrait cutout data. During training, in addition to the loss function, a penalty constraint is also added to the pre-clip factor layer. That is, during backpropagation, when updating the parameters of each layer belonging to the U-net model in the first model, the gradient used is the gradient of the loss function, and when updating the gradient of the pre-clip factor layer in the first model, in addition to the gradient of the loss function, the gradient of the lasso penalty term is also added to constrain the training parameters of the pre-clip factor layer, thereby obtaining a network model with high accuracy.

[0051] It should be noted that the first model is trained using annotated portrait cutout data to obtain a second model for portrait cutout, and then a lightweight model for portrait cutout is obtained through step S140. For the sake of convenience, this embodiment and subsequent embodiments are described using a portrait cutout model based on a U-net model as an example. It should be understood that the exemplary description cannot be interpreted as a specific limitation of the present application. The embodiments of the present application can also be applied to application scenarios of models based on the U-net model for object classification, object recognition, etc.

[0052] S140, pruning channels in the second model whose parameters of the pre-pruned factor layer are less than a threshold, to obtain a third model.

[0053] The channels of the pre-clipping factor layer in the second model whose parameters are less than the threshold are pruned, thereby obtaining a lightweight third model. It should be noted that the present application does not impose any specific restrictions on the specific value of the threshold.

[0054] The larger the parameter of the pre-clipping factor layer, the more sensitive the corresponding channel is to pruning. Conversely, the smaller the parameter of the pre-clipping factor layer, the less sensitive the corresponding channel is to pruning.

[0055] In one embodiment, when the parameter of the pre-pruning factor layer connected to a convolution unit in the second model is less than a threshold, the convolution unit is pruned. Figure 3 As shown in Figure (b), when the parameter of the 1×1 pre-clipping factor layer is less than the threshold, the 3×3 convolution unit in front of it is pruned. When the parameter of the pre-clipping factor layer connected to a convolution unit in the second model is greater than the threshold, the convolution unit is retained. It should be noted that when the parameter of the pre-clipping factor layer connected to a convolution unit in the second model is equal to the threshold, it can be selected and set according to the actual situation, or the convolution unit can be pruned or retained, and this application does not impose specific restrictions on this.

[0056] In one embodiment, channels in the second model whose parameters of the pre-clipping factor layer are less than a threshold are pruned by setting the weight parameters of the pre-clipping factor layer and the convolutional units preceding it to zero. In another embodiment, the pre-clipping factor layer and the convolutional units preceding it can be directly pruned, which can further reduce the amount of computation and further improve the inference speed.

[0057] This embodiment provides a lightweight method for a U-net model, which adds a pre-pruning factor layer to the U-net model and performs pruning based on the trained parameters of the pre-pruning factor layer, thereby achieving lightweighting of the network model and ensuring a high accuracy while accelerating network model inference.

[0058] Figure 4 This is a schematic diagram of the implementation process of a lightweight method of a U-net model provided by another embodiment of the present application. The lightweight method of the U-net model in this embodiment can be executed by an electronic device. Figure 4 As shown, the lightweight method of the U-net model may include steps S210 to S260. It should be noted that this embodiment is different from the aforementioned Figure 1 The similarities of the illustrated embodiments can be found in the above description and will not be repeated here.

[0059] S210, obtaining a U-net model.

[0060] S220: Add a pre-clipping factor layer to the U-net model to obtain a first model.

[0061] S230: Use the first sample data to train the first model to obtain a second model.

[0062] S240: Prune the channels whose parameters of the pre-pruned factor layer in the second model are smaller than a threshold value to obtain a third model.

[0063] S250: If the third model does not meet the preset condition, the threshold is updated and the process returns to execute the above step S240.

[0064] S260: If the third model meets the preset conditions, store the third model.

[0065] In some embodiments, the preset condition may be that one or more of the size, precision, and accuracy of the third model obtained after pruning meets the preset condition. It should be noted that the preset condition can be selected and set according to actual circumstances, and this application does not impose any specific restrictions on this.

[0066] This embodiment provides a lightweight method for a U-net model. Based on the above embodiment, the threshold is dynamically updated so that the third model obtained after pruning meets the preset conditions. This can further improve the adaptability of the lightweight network model. For example, it can further improve the accuracy of the lightweight network model in application scenarios.

[0067] Figure 5 This is a schematic diagram of the implementation process of a lightweight method of a U-net model provided by another embodiment of the present application. The lightweight method of the U-net model in this embodiment can be executed by an electronic device. Figure 5 As shown, the lightweight method of the U-net model may include steps S310 to S370. It should be noted that this embodiment is different from Figure 4 The same points as those in the embodiment shown above are mentioned above and will not be described here again. Figure 4 The improvements based on the embodiment shown are for illustrative purposes only. It should be understood that Figure 4 Modifications to the illustrated embodiment can also be used to Figure 1 The embodiment shown has been improved, and the working principle is the same, so it will not be described in detail.

[0068] S310, obtaining a U-net model.

[0069] S320: Add a pre-clipping factor layer to the U-net model to obtain a first model.

[0070] S330: Use the first sample data to train the first model to obtain a second model.

[0071] S340: Fusing the convolutional units in the second model.

[0072] The convolutional units in the second model are fused to facilitate the fusion of the subsequent pre-clipping factor layer with the convolutional unit before it.

[0073] In some embodiments, the convolutional layer and batch normalization (BN) layer of a convolutional unit are fused. If the BN layer follows the convolutional layer directly, the BN layer can be directly embedded in the convolutional layer's calculations, which is equivalent to scaling the convolution kernel by a certain factor and making certain changes to the bias. Fusion of the BN layer into the convolutional layer is equivalent to modifying the convolution kernel without increasing the computational effort of the convolutional layer, while also eliminating the computational effort of the entire BN layer.

[0074] In some other embodiments, the convolution layer, batch normalization layer, and activation function of the convolution unit are fused.

[0075] S350: Prune the channels whose parameters of the pre-clipping factor layer in the fused second model are smaller than a threshold value to obtain a third model.

[0076] S360: If the third model does not meet the preset condition, the threshold is updated and the process returns to execute the above step S350.

[0077] S370: If the third model meets the preset conditions, the pre-pruned factor layer retained in the third model is fused with the preceding convolution unit to obtain a fourth model.

[0078] In some embodiments, the retained parameters of the pre-clipping factor layer are fused with the preceding convolutional unit to further reduce computational complexity. As a non-limiting example, the kernel of the pre-clipping factor layer is convolved with the transposed kernel of the preceding convolutional unit (i.e., the convolutional layer obtained by fusion of the convolutional units), and then transposed to obtain the structure and parameters of the fused network model.

[0079] This embodiment provides a lightweight method for a U-net model. Based on any of the aforementioned embodiments, the network model is integrated to reduce the amount of calculation and further improve the inference speed of the lightweight network model.

[0080] Figure 6 This is a schematic diagram of the implementation process of a lightweight method of a U-net model provided by another embodiment of the present application. The lightweight method of the U-net model in this embodiment can be executed by an electronic device. Figure 6 As shown, the lightweight method of the U-net model may include steps S410 to S480. It should be noted that this embodiment is different from Figure 5 The same points as those in the embodiment shown above are mentioned above and will not be described here again. Figure 5The improvements based on the embodiment shown are for illustrative purposes only. It should be understood that Figure 5 Modifications to the illustrated embodiment can also be used to Figure 1 or Figure 4 The embodiment shown has been improved, and the working principle is the same, so it will not be described in detail.

[0081] S410, obtaining a U-net model.

[0082] S420: Add a pre-clipping factor layer to the U-net model to obtain a first model.

[0083] S430: Use the first sample data to train the first model to obtain a second model.

[0084] S440: Fusing the convolutional units in the second model.

[0085] S450: Prune the channels whose parameters of the pre-clipping factor layer in the fused second model are smaller than a threshold value to obtain a third model.

[0086] S460: If the third model does not meet the preset condition, the threshold is updated and the process returns to execute the above step S350.

[0087] S470: If the third model meets the preset conditions, the pre-pruned factor layer retained in the third model is fused with the convolution unit preceding it to obtain a fourth model.

[0088] S480: Optimize and train the fourth model using the second sample data to obtain a fifth model.

[0089] The amount of the second sample data is less than the amount of the first sample data. The second sample data is sample data associated with the application scenario.

[0090] In one embodiment, the first sample data is a large amount of annotated portrait cutout data, and the fourth model is a portrait cutout model. The fourth model is tuned using a smaller amount of second sample data that meets the application scenario requirements to avoid accuracy loss in the application scenario after the network model is cropped.

[0091] When the portrait cutout model needs to be deployed in specific application scenarios, such as living rooms, half-body, full-body, or subway stations, it is fine-tuned and trained using labeled portrait cutout data that meets the application scenario requirements. The fine-tuning training process includes: inputting a portrait image, inferring the portrait cutout result through the fourth model, calculating the loss between the portrait cutout result and the manual annotation result, and backpropagating the loss gradient to update the fourth model parameters. After training iterations, the fine-tuned network model, i.e., the fifth model, is obtained.

[0092] After obtaining the optimized network model, the network model can be deployed to electronic devices to achieve end-to-end application.

[0093] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0094] One embodiment of the present application further provides a lightweight device for a U-net model. For details not described in detail in the lightweight device, please refer to the description of the lightweight method embodiment above.

[0095] See also Figure 7 , Figure 7 1 is a schematic diagram of a lightweight device for a U-net model according to an embodiment of the present application. The lightweight device includes an acquisition module 71 , an addition module 72 , a training module 73 , and a pruning module 74 .

[0096] An acquisition module 71 is used to acquire a U-net model;

[0097] An adding module 72 is used to add a pre-clipping factor layer to the U-net model to obtain a first model.

[0098] The training module 73 is used to train the first model using the first sample data to obtain the second model.

[0099] The pruning module 74 is configured to prune the channels in the second model whose parameters of the pre-pruned factor layer are less than a threshold value, to obtain a third model.

[0100] In some embodiments, the adding module 72 is specifically used to add a pre-clipping factor layer after the first convolution unit of each convolution module in the encoder of the U-net model to obtain a first model.

[0101] In some embodiments, during the back propagation process of training the first model using the first sample data to obtain the second model, the parameters of each layer belonging to the U-net model in the first model are updated using the gradient of the loss function, and the parameters of the pre-clipping factor layer in the first model are updated using the gradient of the loss function and the gradient of the penalty term.

[0102] In some embodiments, as Figure 8 As shown, in Figure 7 Based on the embodiment shown, the pruning module 74 is improved and specifically includes a first submodule 741 , a second submodule 742 , and a third submodule 743 .

[0103] The first submodule 741 is configured to prune channels in the second model whose parameters of the pre-clipping factor layer are less than a threshold value, to obtain a third model.

[0104] The second submodule 742 is configured to update the threshold if the third model does not meet the preset condition and return to the first submodule 741;

[0105] The third submodule 743 is configured to store the third model if the third model meets a preset condition.

[0106] In some embodiments, as Figure 9 As shown in Figure 7 As an example of improvement based on the embodiment shown, the lightweight device further includes a fusion module 75. Figure 10 As shown in Figure 8 Taking the embodiment shown as an example for improvement, the lightweight device further includes a fusion module 75. The fusion module 75 is used to fuse the pre-clipping factor layer retained in the third model with the convolution unit in front of it to obtain a fourth model.

[0107] In some embodiments, as Figure 11 As shown in Figure 7 As an example of an improvement based on the illustrated embodiment, the lightweight device further includes a tuning module 76. Tuning module 76 is configured to use the second sample data to perform tuning training on the third model to obtain a fourth model. The amount of the second sample data is less than the amount of the first sample data, and the second sample data is sample data associated with the application scenario.

[0108] In some other embodiments, such as Figure 12 As shown in Figure 10 As an example of an improvement based on the illustrated embodiment, the lightweight device further includes a tuning module 76. Tuning module 76 is configured to train the fourth model using the second sample data to obtain a fifth model. The second sample data is smaller in quantity than the first sample data, and the second sample data is sample data associated with the application scenario.

[0109] In some other embodiments, Figure 8 or Figure 9 Based on the embodiment shown, the lightweight device is improved and further includes a tuning module 75 .

[0110] In some embodiments, based on any of the foregoing embodiments, the first sample data is labeled portrait cutout data, and the third model is a model for portrait cutout.

[0111] An embodiment of the present application further provides an electronic device, such as Figure 13 As shown, the electronic device may include one or more processors 100 ( Figure 13Only one is shown in the figure), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a lightweight program for a U-net model. When one or more processors 100 execute computer program 102, each step in the embodiment of the lightweight method for the U-net model can be implemented. Alternatively, when one or more processors 100 execute computer program 102, the functions of each module / unit in the embodiment of the lightweight device for the U-net model can be implemented, which is not limited here.

[0112] Those skilled in the art will understand that Figure 13 The electronic device is merely an example and does not limit the electronic device. The electronic device may include more or fewer components than shown in the figure, or may combine certain components or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0113] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0114] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Furthermore, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the electronic device. The memory 101 may also be used to temporarily store data that has been output or is about to be output.

[0115] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0116] An embodiment of the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the steps in the embodiment of the lightweight method of the U-net model can be implemented.

[0117] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps of the embodiment of the lightweight method of the U-net model.

[0118] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0119] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0121] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0122] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0123] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned method embodiment, and the computer program that can be completed by instructing the relevant hardware through a computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0124] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A lightweight method for a U-net model, characterized in that: Executed by electronic devices for portrait cutout, including: Obtain a portrait cutout model based on the U-net model; Adding a pre-clipping factor layer to the U-net model to obtain a first model; comprising: Adding a pre-clipping factor layer after the first convolution unit of each convolution module in the encoder of the U-net model to obtain a first model; The first model is trained using the labeled portrait cutout data as first sample data to obtain a second model for portrait cutout; during training, in addition to the loss function, a penalty constraint is added to the pre-cropping factor layer to constrain the training parameters of the pre-cropping factor layer; in the back-propagation process of training the first model using the first sample data to obtain the second model, the parameters of each layer belonging to the U-net model in the first model are updated using the gradient of the loss function, and the parameters of the pre-cropping factor layer in the first model are updated using the gradient of the loss function and the gradient of the penalty term; Pruning channels whose parameters of the pre-clipping factor layer in the second model are less than a threshold to obtain a third model, where the third model is a lightweight model for portrait cutout; The pruning of channels whose parameters of the pre-clipping factor layer in the second model are less than a threshold includes: setting the weight parameters of the pre-clipping factor layer and the convolution unit in front of the pre-clipping factor layer to zero, and / or directly pruning the pre-clipping factor layer and the convolution unit in front of the pre-clipping factor layer.

2. The lightweight method according to claim 1, wherein: The step of pruning channels in the second model whose parameters of the pre-clipping factor layer are less than a threshold value to obtain a third model comprises: Step S1, pruning the channels whose parameters of the pre-pruned factor layer in the second model are less than a threshold, to obtain a third model, Step S2: if the third model does not meet the preset condition, update the threshold and return to execute the above step S1; Step S3: If the third model meets the preset conditions, the third model is stored.

3. The lightweight method according to claim 1, wherein: After pruning the channels whose parameters of the pre-clipping factor layer in the second model are less than a threshold to obtain the third model, the method further includes: fusing the pre-clipping factor layer retained in the third model with the convolution unit in front of it to obtain the fourth model.

4. The lightweight method according to claim 1, wherein: Also includes: The third model is tuned and trained using second sample data to obtain a fourth model, where the amount of the second sample data is less than the amount of the first sample data, and the second sample data is sample data associated with the application scenario.

5. A lightweight device for a U-net model, characterized in that: include: The acquisition module is used to obtain the portrait cutout model based on the U-net model; An adding module, for adding a pre-clipping factor layer to the U-net model to obtain a first model, comprising: Adding a pre-clipping factor layer after the first convolution unit of each convolution module in the encoder of the U-net model to obtain a first model; A training module, configured to use the labeled portrait cutout data as first sample data to train the first model to obtain a second model for portrait cutout; during training, in addition to the loss function, a penalty constraint is added to the pre-cropping factor layer to constrain the training parameters of the pre-cropping factor layer; in a back-propagation process of training the first model using the first sample data to obtain the second model, the parameters of each layer belonging to the U-net model in the first model are updated using the gradient of the loss function, and the parameters of the pre-cropping factor layer in the first model are updated using the gradient of the loss function and the gradient of the penalty term; a pruning module, configured to prune channels in the second model whose parameters of the pre-clipping factor layer are less than a threshold, to obtain a third model, wherein the third model is a lightweight model for portrait cutout; The pruning of channels whose parameters of the pre-clipping factor layer in the second model are less than a threshold includes: setting the weight parameters of the pre-clipping factor layer and the convolution unit in front of the pre-clipping factor layer to zero, and / or directly pruning the pre-clipping factor layer and the convolution unit in front of the pre-clipping factor layer.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the lightweight method according to any one of claims 1 to 4 is implemented.

7. A computer storage medium storing a computer program, wherein: When the computer program is executed by a processor, the lightweight method according to any one of claims 1 to 4 is implemented.

Images