Image processing model training method, live image processing method and device

Abstract

This application relates to an image processing model training method, a live image processing method, and an apparatus. The method involves obtaining a first loss function of the image processing model to be trained, the weights of the normalized layer, and the variance corresponding to the weights. Based on the weights and variance, a second loss function is constructed to restrict the weights of the normalized layer of the image processing model to a target range. A target loss function is then constructed based on the first and second loss functions of the image processing model before quantization-perceptual training. Based on the target loss function, the image processing model is quantized-perceptually trained using training image data. This avoids the abnormal weight distribution of the convolutional layer affecting the quantization effect during quantization-perceptual training, thereby improving the accuracy of quantization-perceptual training of the image processing model.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing model training method, a live image processing method, and an apparatus. Background Technology

[0002] With the development of artificial intelligence technology, deep learning-based image processing has been widely applied in the live streaming field to achieve image processing during live broadcasts. Existing image processing models often require a lot of computing resources. In order to make image processing models more applicable to devices with limited computing resources, model quantization is usually used to compress the image processing models. Model quantization refers to approximating floating-point activation values ​​or weights (usually represented as 32-bit floating-point numbers) as low-bit integers (16 bits or 8 bits), and then completing the calculation in low-bit representation. By quantizing the model, the computing resources used by the model can be reduced, allowing the model to be better applied to devices with limited computing resources.

[0003] Quantization-aware training is a common model compression method. By introducing pseudo-quantization during training, the model is fine-tuned to adapt to the errors introduced by quantization, thereby improving the accuracy of model training.

[0004] However, in existing quantization-aware training processes, since the variance of the normalization layer statistics of the model may be a very small number, the weights of the convolutional layer may become very large after the convolutional layer and the normalization layer are fused, thus affecting the accuracy of the quantization model. Summary of the Invention

[0005] Based on this, the purpose of this application is to provide an image processing model training method, a live image processing method and apparatus, which can effectively improve the training accuracy of the image processing model.

[0006] According to a first aspect of the embodiments of this application, an image processing model training method is provided, the image processing model training method comprising:

[0007] The training image data, the first loss function of the image processing model to be trained, the weights of the normalization layer, and the variance corresponding to the weights are obtained. A second loss function is constructed based on the weights and the variance. The first loss function is the loss function of the image processing model when it is not trained by quantization perception. The second loss function is used to limit the weights of the normalization layer of the image processing model to be within a target range.

[0008] Construct a target loss function based on the first loss function and the second loss function;

[0009] Based on the target loss function, the image processing model is trained using the training image data through quantitative perception training.

[0010] According to a second aspect of the embodiments of this application, a live image processing method is provided, comprising the following steps:

[0011] Acquire live stream images;

[0012] A key point detection model is constructed based on the image processing model trained by any of the above image processing model training methods, and the key point detection model is used to obtain key point data of live images.

[0013] Based on the key point data, key points are identified on the live stream image, and a live stream image including the key points is displayed.

[0014] According to a third aspect of the embodiments of this application, an image processing model training apparatus is provided, the image processing model training apparatus comprising:

[0015] The data acquisition module is used to acquire training image data, a first loss function of the image processing model to be trained, the weights of the normalization layer, and the variance corresponding to the weights, and to construct a second loss function based on the weights and the variance; wherein, the first loss function is the loss function of the image processing model when it is not trained by quantization perception, and the second loss function is used to limit the weights of the normalization layer of the image processing model to be within a target range.

[0016] The target loss function acquisition module is used to construct a target loss function based on the first loss function and the second loss function;

[0017] The training module is used to perform quantitative perception training on the image processing model using the training image data based on the target loss function.

[0018] According to a fourth aspect of the embodiments of this application, a live streaming image processing apparatus is provided, comprising:

[0019] The live image acquisition module is used to acquire live images;

[0020] The key point data acquisition module is used to construct a key point detection model based on the image processing model trained by any of the image processing model training methods described above, and to acquire key point data of live images using the key point detection model.

[0021] The identification module is used to identify key points on the live image based on the key point data, and to display the live image including the key points.

[0022] According to a fifth aspect of the present application, an electronic device is provided, comprising: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the method described in any one of the preceding embodiments.

[0023] According to a sixth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the method described in any one of the above-described embodiments.

[0024] In this application, a first loss function, the weights of the normalized layer, and the variance corresponding to the weights of the image processing model to be trained are obtained. A second loss function is constructed based on the weights and variance to restrict the weights of the normalized layer of the image processing model to be within a target range. Thus, a target loss function is constructed based on the first and second loss functions of the image processing model before quantization-aware training. Based on the target loss function, the image processing model is quantized-aware trained using training image data. This avoids the abnormal weight distribution of the convolutional layer affecting the quantization effect during quantization-aware training and improves the accuracy of quantization-aware training of the image processing model.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application.

[0026] To better understand and implement this application, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0027] Figure 1 A schematic diagram illustrating the application environment of a method provided in one embodiment of this application;

[0028] Figure 2 A flowchart illustrating an image processing model training method provided in one embodiment of this application;

[0029] Figure 3 A flowchart of step S130 provided in one embodiment of this application;

[0030] Figure 4 A flowchart illustrating a live image processing method provided in one embodiment of this application;

[0031] Figure 5 A schematic diagram of a live image provided in one embodiment of this application;

[0032] Figure 6 This is a schematic diagram of the structure of an image processing model training device provided in one embodiment of this application;

[0033] Figure 7A schematic diagram of the structure of a live image processing apparatus provided in one embodiment of this application;

[0034] Figure 8 This is a schematic block diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0037] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0038] In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. The singular forms "a," "the," and "the" used in this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. The words "if" or "when" as used herein can be interpreted as "when," "in response to a determination." Furthermore, in the description of this application, unless otherwise stated, "a plurality" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0039] Please see Figure 1 , Figure 1 This diagram illustrates an application scenario of a method provided in one embodiment of this application. The application scenario includes a broadcaster 110, a server 120, and a server 130 provided in this embodiment. The broadcaster 110 and the server 130 interact through the server 120.

[0040] The broadcaster terminal 110 refers to the terminal that sends the live video, which is usually the viewer terminal used by the broadcaster in the live broadcast.

[0041] The server-side 130 refers to the end that receives and watches live online videos, which is usually the viewer end used by viewers watching videos in a live online broadcast.

[0042] The hardware referred to by the broadcaster terminal 110 and the server terminal 130 essentially refers to computer equipment, specifically, such as... Figure 1 As shown, it can be a computer device such as a smartphone, a smart interactive whiteboard, or a personal computer. Both the broadcaster terminal 110 and the server terminal 130 can access the Internet through known network access methods and establish a data communication link with the server 10.

[0043] The server 120, acting as a service server, can further connect to relevant audio data servers, video streaming servers, and other servers providing related support, thereby forming a logically interconnected service cluster to serve relevant terminal devices, such as… Figure 1 The broadcaster 110 and server 130 shown provide services.

[0044] This application discloses an image processing model training method, which can be applied to the aforementioned server 120. The image processing model training method provided in this application will be described in detail below with reference to the accompanying drawings.

[0045] Please see Figure 2 This application provides an image processing model training method, including the following steps:

[0046] S110: Obtain training image data, the first loss function of the image processing model to be trained, the weights of the normalization layer, and the variance corresponding to the weights; construct a second loss function based on the weights and the variance.

[0047] Wherein, the first loss function is the loss function of the image processing model when it is not trained by quantization perception, and the second loss function is used to limit the weights of the normalization layer of the image processing model to be within the target range;

[0048] S120: Construct a target loss function based on the first loss function and the second loss function;

[0049] S130: Based on the target loss function, the image processing model is subjected to quantitative perception training using the training image data.

[0050] In step S110, the training image data can be live image data. Specifically, the training image data can include multiple live images labeled with key point information.

[0051] Image processing models can be built based on deep learning algorithms such as convolutional neural network models. Image processing models can be used to implement image processing tasks such as key point detection in live images. Image processing models can include at least one convolutional layer and at least one normalization layer. The convolutional layer is used to perform convolution operations on the input data; the normalization layer normalizes the output data of the convolutional layer.

[0052] The image processing model of this application can be a model obtained by pre-training using training image data, and the first loss function is the loss function of the image processing model pre-training process.

[0053] The target range is used to limit the numerical range of the weights of the normalization layer in the image processing model. The target range can be a numerical range determined by the maximum and minimum values, or it can be a numerical range determined by a target threshold set by the user. For example, the target range can be less than the target threshold.

[0054] In one embodiment, the image processing model can be constructed based on a convolutional neural network algorithm, and the image processing model includes a first pseudo-quantization node, a second pseudo-quantization node, a convolutional layer, and a normalization layer.

[0055] The first pseudo-quantization node is used to perform pseudo-quantization on the activation values ​​of each layer of the image processing model; the second pseudo-quantization node is used to perform pseudo-quantization on the trainable parameters of each layer of the image processing model; the convolutional layer is used to perform convolution operations on the input data; and the normalization layer normalizes the output data of the convolutional layer.

[0056] The first and second pseudo-quantization nodes can be used to statistically analyze the maximum and minimum values ​​of the data flowing through the node during training and participate in forward propagation, increasing the value of the loss function. The optimizer senses this increase in loss value and performs continuous backpropagation learning to improve the accuracy of the model.

[0057] In this application, the first pseudo-quantization node is used to perform pseudo-quantization on the activation values ​​of each layer of the image processing model, and the second pseudo-quantization node is used to perform pseudo-quantization on the trainable parameters of each layer of the image processing model, thereby reducing the accuracy loss of the model after quantization and improving the accuracy of the model.

[0058] The activation value is the output of the neuron in the previous layer of the convolutional neural network.

[0059] Trainable parameters are parameters that can be adjusted during the training of an image processing model, including but not limited to the model's weights, biases, and other parameters.

[0060] Specifically, in the embodiments of this application, the first pseudo-quantization node includes a first quantization unit and a first dequantization unit;

[0061] The first quantization unit is used to quantize the input activation value; the first dequantization unit is used to dequantize the output data of the quantization unit.

[0062] The first quantization unit can use existing quantization calculation formulas to quantize the activation value into a value converted to a specified number of bits.

[0063] Specifically, in this embodiment of the application, the first quantization unit performs a quantization operation on the input activation value, specifically including:

[0064] The activation value of the input is rounded using a rounding function, and the rounded activation value is then restricted to a target range using a truncation function.

[0065] The rounding function can be the Round function, which is used to round the activation value to the specified number of decimal places.

[0066] The truncation function can be a Clip function. The truncation function is used to restrict the rounded activation value to a target range. When the rounded activation value is greater than the target range, the activation value is changed to the maximum value of the target range. When the rounded activation value is less than the target range, the activation value is changed to the minimum value of the target range.

[0067] The target range can be set according to the user's needs. In this embodiment, the target range can be set to... This represents the minimum value when quantized to b bits. This represents the maximum value when quantized to b bits.

[0068] The first quantization unit in this application can be used to perform symmetric quantization on the input activation values. Symmetric quantization means that the maximum and minimum values ​​within the target range have opposite signs and equal absolute values, that is... This application first uses a rounding function to round the activation value, and then uses a truncation function to restrict the rounded activation value to a target range, thereby limiting the activation value to the target range and realizing the quantification of the activation value.

[0069] Preferably, in one embodiment, the input activation value is quantized in the following manner:

[0070]

[0071] Where X represents the input activation value, and S represents the training parameters. This represents the minimum value when quantized to b bits. This indicates the maximum value when quantized to b bits. Round represents the rounding function, and Cliip represents the truncation function. In this application, it indicates truncating the data to... and In between, Qunt(X) represents the output data of the first quantization unit;

[0072] In this application, the training parameter S can be a value less than 1. This application makes it easier to implement the rounding function by dividing the input activation value X by the training parameter S, thereby making it easier to quantize the activation value.

[0073] The first dequantization unit can perform dequantization on the output data of the quantization unit using existing dequantization calculation formulas, specifically including:

[0074] The output data of the first quantization unit is dequantized in the following manner:

[0075] X'=DeQunt(Qunt(X))=Qunt(X)*S

[0076] Where X′ represents the output data of the first dequantization unit.

[0077] The second pseudo-quantization node includes a second quantization unit and a second dequantization unit;

[0078] The second quantization unit is used to quantize the trainable parameters; the second dequantization unit is used to dequantize the output data of the quantization unit.

[0079] The quantization operation method of the second quantization unit is the same as that of the first quantization unit. You can refer to the quantization operation process of the first quantization unit. The dequantization operation method of the second dequantization unit is the same as that of the first dequantization unit. You can refer to the dequantization operation process of the first dequantization unit. It will not be repeated here.

[0080] In step S120, the objective loss function is used to measure the degree of difference between the model's predicted value and the true value. Generally, the smaller the objective loss function, the better the robustness of the model.

[0081] During the model training phase, after the training data is fed into the model, the predicted values ​​are output through forward propagation. The difference between the predicted and true values ​​is calculated based on the loss function, which is the loss value. After obtaining the loss value, the model updates its parameters through backpropagation to reduce the loss between the true and predicted values, making the model's predicted values ​​move closer to the true values, thereby achieving the learning objective.

[0082] like Figure 3 As shown, the image processing model is trained using the training image data through quantitative perception training, specifically including:

[0083] S131: Merge the normalization layer and the convolutional layer to obtain the target weights of the fused convolutional layer;

[0084] S132: Delete the pseudo-quantization nodes and normalization layers of the image processing model, reconstruct the image processing model based on the target weights and the target loss function, and perform quantization-aware training on the image processing model using the training data.

[0085] In step S131, fusing the normalization layer and the convolutional layer can be achieved by merging the convolution and normalization operations of the normalization layer and the convolutional layer, fusing the weights of the normalization layer into the weights of the convolutional layer, using the fused weights as the weights of the convolutional layer, and simultaneously deleting the normalization layer to reduce I / O operations and improve model performance.

[0086] The weights and biases of the fused convolutional layers can be obtained from the convolution calculation formula of the convolutional layers before fusion and the normalization calculation formula of the normalization layers.

[0087] Specifically, the step of fusing the normalization layer and the convolutional layer includes:

[0088] Obtain the weights of the convolutional layer, the bias of the convolutional layer, the weights of the normalized layer, the bias of the normalized layer, the mean of the weights of the normalized layer, and the variance of the weights of the normalized layer.

[0089] Obtain the target weights of the fused convolutional layer in the following manner:

[0090]

[0091] Among them, W c ′ onv W represents the target weights of the fused convolutional layer. conv W represents the weights of the convolutional layer before fusion. bn Var represents the weights of the normalization layer. bn This represents the variance of the weights in the normalized layer.

[0092] Obtain the bias values ​​of the fused convolutional layer as follows:

[0093]

[0094] Among them, B c ′ onvB represents the bias value of the fused convolutional layer. conv Mean represents the bias value of the convolutional layer before fusion. bn B represents the mean of the weights of the normalized layer. bn This represents the bias value of the normalized layer before fusion.

[0095] Since the range of weight values ​​after fusion of the weights of the convolutional layer and the batch normalization layer increases when the variance of the batch normalization layer statistics is relatively small, which affects the quantization effect, this application sets a second loss function to limit the weights of the normalization layer of the image processing model to be within the target range.

[0096] The second loss function can specifically limit the range of variance or standard deviation of the weights in the normalization layer of the image processing model, thereby keeping the weights within a target range. When limiting the range of variance of the image normalization layer weights, a regularization factor can be added to the parameters of the normalization layer. This regularization factor can be used to perform L2 regularization on the parameters of the normalization layer. Compared to L1 regularization, L2 regularization is more computationally efficient, can prevent overfitting, and improve model accuracy.

[0097] Alternatively, in another embodiment, the accuracy of the quantized image processing model can be improved by limiting the ratio of the normalized layer weights to their statistical standard deviation to a target range, thereby making the weight distribution after fusion more reasonable.

[0098] Specifically, the second loss function can be a function used to limit the ratio of the absolute value of the weights of the normalization layer of the image processing model to the standard deviation of the corresponding weights to be less than a target threshold, that is:

[0099]

[0100] Among them, W bn Var represents the weights of the normalization layer. bn θ represents the variance of the weights of the normalization layer, and θ represents the target threshold, which can be any constant, and can be set to 5 in this application.

[0101] The second loss function can be set as:

[0102]

[0103] Preferably, the second loss function can be:

[0104]

[0105] Among them, L bn Let W represent the second loss function. bnVar represents the weights of the normalization layer. bn θ represents the variance of the weights of the normalization layer, and θ represents the target threshold.

[0106] The larger the loss value, the more this application uses... The second loss function is constructed using the logarithm, compared to directly using... To construct a second loss function, the smaller the loss value, the easier it is for the model to optimize in the target direction during training, which can improve the training efficiency of the model.

[0107] Preferably, this application can select a portion of the normalized layers whose model weight variance is less than a specified threshold to construct the second loss function, thereby reducing the amount of training data and improving training efficiency. The specified threshold can be set according to user needs, for example, it can be set to 0.01.

[0108] Specifically, the step of constructing a target loss function based on the first loss function and the second loss function includes: constructing a target loss function based on the sum of the product of the second loss function and the training parameters and the first loss function.

[0109] The training parameters are used to balance the first loss function and the second loss function to prevent the target loss function from being dominated by the second loss function, which would affect the training accuracy of the model.

[0110] Construct the target loss function as follows:

[0111] L = L task +α*L bn

[0112] Where L represents the target loss function, L task Let L represent the first loss function. bn Let α represent the second loss function, and let α represent the training parameter. The training parameter can be set according to the user's training needs; for example, in this application, it can be set to 0.2. In step S130, based on the above target loss function, the image processing model is subjected to quantization perception training using training image data. When the image processing model meets the training completion condition, it is determined that the image processing model has completed quantization perception training.

[0113] The training completion condition can be set according to user needs. For example, in this application, the training completion condition can be that the number of iterations of the image processing model reaches the target number of iterations.

[0114] The image processing model trained by this application can compress the size of the original image processing model and improve its computational efficiency, thereby ensuring that the model can be deployed on devices with relatively efficient computing resources, such as mobile devices. This allows the device to run the image processing model to perform corresponding image processing, making it convenient for broadcasters to run the image processing model on mobile devices for live streaming slimming and other processing, enriching the broadcaster's live streaming features. At the same time, it can ensure the quantization accuracy of the model and improve the accuracy and efficiency of image processing.

[0115] In this embodiment, by obtaining the first loss function of the image processing model to be trained, the weights of the normalized layer, and the variance corresponding to the weights, a second loss function is constructed based on the weights and variance to restrict the weights of the normalized layer of the image processing model to be within a target range. Thus, a target loss function is constructed based on the first and second loss functions of the image processing model before quantization-aware training. Based on the target loss function, the image processing model is quantized-aware trained using training image data. This avoids excessive differences in the weight distribution of the convolutional layers during quantization-aware training, which could lead to a decrease in the accuracy of the quantization model and affect the quantization effect, thereby improving the accuracy of the quantization-aware training of the image processing model. Simultaneously, this application addresses the problem of large numerical differences after the fusion of the normalized layer weights and the convolutional layer weights by using the first and second loss functions to construct the target loss function. This allows the weight distribution to be adaptively optimized during training, resulting in an image processing model that is easy to deploy with quantization.

[0116] Please see Figure 4 This application also provides a live image processing method, including the following steps:

[0117] S210: Acquire live stream image;

[0118] A live stream image is a screenshot captured from a live stream. In one embodiment, a live stream image can also be a video frame image from a live stream video.

[0119] Specifically, the steps for acquiring live images include:

[0120] In response to an image processing instruction, a video frame image of the live video stream is acquired, and the video frame image is used as the live image.

[0121] Image processing instructions can be control instructions issued by the host or viewers in the live broadcast room. After receiving the image processing instruction, the live broadcast image in the current live broadcast room is processed accordingly according to the image processing instruction. Image processing includes, but is not limited to, image deformation processing such as slimming, face slimming, and leg stretching.

[0122] S220: Construct a key point detection model based on the image processing model trained by any of the above image processing model training methods, and use the key point detection model to obtain key point data of live images;

[0123] Specifically, the image processing model obtained after training in the above embodiments can be directly used as the key point detection model, or, based on the above image processing model, the above image processing model can be trained using several live images containing key point annotation data to obtain the key point detection model.

[0124] Key points can be human body key points or facial key points, and the specific settings can be set according to the different live images and the user's detection needs.

[0125] Keypoint detection models are used to detect keypoint data in images. Keypoint data can include the number and location of keypoints.

[0126] S230: Based on the key point data, identify key points on the live image and display the live image including the key points.

[0127] In one embodiment, the method further includes the following steps:

[0128] Based on the key point data, the live image is subjected to deformation processing.

[0129] Image distortion processing includes, but is not limited to, slimming, face slimming, and leg stretching. The specific settings can be configured according to the content of the live stream image and user needs.

[0130] By deforming live stream images using key point data, the live streamer's interest and the user's viewing experience can be enhanced.

[0131] Please see Figure 5 This is a schematic diagram of a live image in one embodiment; as shown below. Figure 5 As shown, the key point data includes human body key point data. After obtaining human body key point data in the live image using the key point detection model, the detected human body key points are marked on the live image, so as to facilitate subsequent deformation processing of the human body such as leg stretching and slimming based on the human body key points.

[0132] In this embodiment, an image processing model trained using the above-described image processing model training method is used to construct a key point detection model. The key point detection model is then used to detect key data in the live stream image, thereby marking the detected key points on the live stream image and visualizing the key points, making it easier for users to perform subsequent processing on the key points.

[0133] Alternatively, the key point data includes facial key point data, and after acquiring the key point data of the live image, the following steps are also included:

[0134] Based on the facial landmark data, facial landmarks are marked on the live stream image, and the live stream image marked with the facial landmarks is displayed.

[0135] After obtaining facial key point data in a live image using a key point detection model, the detected facial key points are marked on the live image, which facilitates subsequent facial contouring such as face slimming based on the facial key points.

[0136] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an image processing model training device disclosed in an embodiment of this application. The device includes:

[0137] The data acquisition module 210 is used to acquire training image data, a first loss function of the image processing model to be trained, the weights of the normalization layer, and the variance corresponding to the weights, and to construct a second loss function based on the weights and the variance; wherein, the first loss function is the loss function of the image processing model when it is not trained by quantization perception, and the second loss function is used to limit the weights of the normalization layer of the image processing model to be within a target range.

[0138] The target loss function acquisition module 220 is used to construct a target loss function based on the first loss function and the second loss function;

[0139] The training module 230 is used to perform quantitative perception training on the image processing model using the training image data based on the target loss function.

[0140] In one embodiment, the second loss function is used to limit the ratio of the absolute value of the weights of the normalization layer of the image processing model to the standard deviation of the weights to be less than a target threshold.

[0141] In one embodiment, the image processing model includes a first pseudo-quantization node, a second pseudo-quantization node, a convolutional layer, and a normalization layer;

[0142] The first pseudo-quantization node is used to perform pseudo-quantization on the activation values ​​of each layer of the image processing model; the second pseudo-quantization node is used to perform pseudo-quantization on the trainable parameters of each layer of the image processing model; the convolutional layer is used to perform convolution operations on the input data; and the normalization layer normalizes the output data of the convolutional layer.

[0143] The training module 230 includes:

[0144] A fusion unit is used to fuse the normalization layer and the convolutional layer to obtain the target weights of the fused convolutional layer.

[0145] The training unit is used to remove pseudo-quantization nodes and normalization layers of the image processing model, reconstruct the image processing model based on the target weights and the target loss function, and perform quantization-aware training on the image processing model using the training data.

[0146] In one embodiment, the first pseudo-quantization node includes a first quantization unit and a first dequantization unit;

[0147] The first quantization unit is used to quantize the input activation value; the first dequantization unit is used to dequantize the output data of the quantization unit.

[0148] In one embodiment, the first quantization unit performs a quantization operation on the input activation value, specifically including:

[0149] The input activation value is quantized in the following way:

[0150]

[0151] Where X represents the input activation value, and S represents the training parameters. This represents the minimum value when quantized to b bits. This represents the maximum value when quantized to b bits, round represents the rounding function, Clip represents the truncation function, and Qunt(X) represents the output data of the first quantization unit;

[0152] The first dequantization unit performs a dequantization operation on the output data of the quantization unit, specifically including:

[0153] The output data of the first quantization unit is dequantized in the following manner.

[0154] X ′ =DeQunt(Qunt(X))=Qunt(X)*

[0155] Where X′ represents the output data of the first dequantization unit.

[0156] The fusion unit includes:

[0157] The data acquisition unit is used to acquire the weights of the convolutional layer, the weights of the normalized layer, and the variance of the weights of the normalized layer before fusion.

[0158] The target weight acquisition unit is used to acquire the target weights of the fused convolutional layer in the following manner:

[0159]

[0160] Among them, W c ′ onv W represents the target weights of the fused convolutional layer. conv W represents the weights of the convolutional layer before fusion. bn Var represents the weights of the normalization layer. bn This represents the variance of the weights in the normalized layer.

[0161] In one embodiment, the target loss function acquisition module 220 constructs the target loss function in the following manner;

[0162] L = L task +α*L bn

[0163] Where L represents the target loss function, L task Let L represent the first loss function. bn Let represent the second loss function, and α represent the training parameters.

[0164] It should be noted that the image processing model training device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the image processing model training method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing model training device provided in the above embodiments and the image processing model training method in the above embodiments belong to the same concept, and its implementation process is detailed in the method embodiments, which will not be repeated here.

[0165] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a live image processing apparatus disclosed in an embodiment of this application. The apparatus includes:

[0166] The live image acquisition module 310 is used to acquire live images;

[0167] The key point data acquisition module 320 is used to construct a key point detection model based on the image processing model trained by the image processing model training method described above, and to acquire key point data of the live image using the key point detection model.

[0168] The identification module 330 is used to identify key points on the live image based on the key point data, and to display the live image including the key points.

[0169] In one embodiment, the live image processing apparatus further includes:

[0170] The deformation processing module is used to perform deformation processing on the live image based on the key point data.

[0171] In one embodiment, the live image acquisition module 310 responds to an image processing instruction to acquire video frame images of the live video stream and uses the video frame images as the live image.

[0172] It should be noted that the live image processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when executing the live image processing method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the live image processing apparatus provided in the above embodiments and the live image processing method in the above embodiments belong to the same concept, and its implementation process is detailed in the method embodiments, which will not be repeated here.

[0173] This embodiment provides an electronic device that can be used to perform all or part of the steps of the methods described in the above embodiments of this application. For details not disclosed in this embodiment, please refer to the method embodiments of this application.

[0174] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 400 may be, but is not limited to, a combination of one or more devices such as servers, personal computers, laptops, smartphones, and tablets.

[0175] In a preferred embodiment of this application, the electronic device 400 includes a memory 410, at least one processor 420, at least one communication bus 430, and a transceiver 440.

[0176] Those skilled in the art should understand that Figure 8 The structure of the electronic device shown does not constitute a limitation of the embodiments of this application. It can be a bus structure or a star structure. The electronic device 400 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0177] In some embodiments, the electronic device 400 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital processors, and embedded devices. The electronic device 400 may also include client devices, including, but not limited to, any electronic product capable of human-computer interaction with a client via a keyboard, mouse, remote control, touchpad, or voice control device, such as personal computers, tablet computers, smartphones, and digital cameras.

[0178] It should be noted that the electronic device 400 is merely an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0179] In some embodiments, the memory 410 stores a computer program that, when executed by the at least one processor 420, implements all or part of the steps in the method as described in the embodiments. The memory 410 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0180] In some embodiments, the at least one processor 420 is the control unit of the electronic device 400, connecting various components of the electronic device 400 via various interfaces and lines. It executes programs or modules stored in the memory 410 and calls data stored in the memory 410 to perform various functions and process data of the electronic device 400. For example, when the at least one processor 420 executes a computer program stored in the memory, it implements all or part of the steps described in the embodiments of this application; or it implements all or part of the functions of the device. The at least one processor 420 may be composed of integrated circuits, such as a single-packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0181] In some embodiments, the at least one communication bus 430 is configured to enable communication between the memory 410 and the at least one processor 420, etc.

[0182] The electronic device 400 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0183] This embodiment provides a computer-readable storage medium storing a computer program thereon. The instructions are adapted to be loaded by a processor and executed by the method of this application embodiment. For the specific execution process, please refer to the detailed description of the above embodiments, which will not be repeated here.

[0184] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0185] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0187] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image processing model training method, characterized in that, Includes the following steps: The training image data, the first loss function of the image processing model to be trained, the weights of the normalization layer, and the variance corresponding to the weights are obtained. A second loss function is constructed based on the weights and the variance. The first loss function is the loss function of the image processing model when it is not trained by quantization perception. The second loss function is used to limit the weights of the normalization layer of the image processing model to be within a target range, and the second loss function is used to limit the ratio of the absolute value of the weights of the normalization layer of the image processing model to the standard deviation corresponding to the weights to be less than a target threshold. Construct a target loss function based on the first loss function and the second loss function; Based on the target loss function, the image processing model is trained using the training image data through quantitative perception training.

2. The image processing model training method of claim 1, wherein, The image processing model includes a first pseudo-quantization node, a second pseudo-quantization node, a convolutional layer, and a normalization layer. The first pseudo-quantization node is used to perform pseudo-quantization on the activation values ​​of each layer of the image processing model; the second pseudo-quantization node is used to perform pseudo-quantization on the trainable parameters of each layer of the image processing model; the convolutional layer is used to perform convolution operations on the input data. The normalization layer normalizes the output data of the convolutional layer; The image processing model is trained using the training image data through quantitative perception training, specifically including: The normalization layer and the convolutional layer are fused to obtain the target weights of the fused convolutional layer; The pseudo-quantization nodes and normalization layers of the image processing model are deleted, and the image processing model is reconstructed based on the target weights and the target loss function. The image processing model is then trained using the training data for quantization perception.

3. The image processing model training method of claim 2, wherein, The first pseudo-quantization node includes a first quantization unit and a first dequantization unit; The first quantization unit is used to quantize the input activation value; the first dequantization unit is used to dequantize the output data of the quantization unit.

4. The image processing model training method of claim 3, characterized in that, The first quantization unit performs a quantization operation on the input activation value, specifically including: The activation value of the input is rounded using a rounding function, and the rounded activation value is then restricted to a target range using a truncation function.

5. The image processing model training method of claim 1, wherein, The steps for constructing the target loss function based on the first loss function and the second loss function specifically include: The target loss function is constructed by combining the product of the second loss function and the training parameters with the sum of the first loss function.

6. A live image processing method, characterized by, Includes the following steps: Acquire live stream images; A key point detection model is constructed based on the image processing model trained by the image processing model training method as described in any one of claims 1-5, and the key point detection model is used to obtain key point data of live images. Based on the key point data, key points are identified on the live stream image, and a live stream image including the key points is displayed.

7. The live image processing method of claim 6, wherein, It also includes the following steps: Based on the key point data, the live image is subjected to deformation processing.

8. The live image processing method according to claim 6, characterized in that, The steps for acquiring live images specifically include: In response to an image processing instruction, a video frame image of the live video stream is acquired, and the video frame image is used as the live image.

9. An image processing model training device, characterized in that, The device includes: The data acquisition module is used to acquire training image data, a first loss function of the image processing model to be trained, the weights of the normalization layer, and the variance corresponding to the weights, and to construct a second loss function based on the weights and the variance; wherein, the first loss function is the loss function of the image processing model when it is not trained by quantization perception, and the second loss function is used to limit the weights of the normalization layer of the image processing model to be within a target range, and the second loss function is used to limit the ratio of the absolute value of the weights of the normalization layer of the image processing model to the standard deviation corresponding to the weights to be less than a target threshold; The target loss function acquisition module is used to construct a target loss function based on the first loss function and the second loss function; The training module is used to perform quantitative perception training on the image processing model using the training image data based on the target loss function.

10. A live streaming image processing device, characterized in that, include: The live image acquisition module is used to acquire live images; The key point data acquisition module is used to construct a key point detection model based on the image processing model trained by the image processing model training method as described in any one of claims 1-5, and to acquire key point data of the live image using the key point detection model. The identification module is used to identify key points on the live image based on the key point data, and to display the live image including the key points.

11. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed as described in any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images