Image processing method, device, equipment and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and provides an image processing method, device and equipment and a storage medium, which are used for reducing the calculation amount and high memory occupation of image processing and lowering the calculation efficiency. The image processing method comprises the following steps: inserting a target quantization node into a pre-trained image processing model to obtain a modified image processing model; performing quantization training on the modified image processing model to obtain a quantized image processing model; deleting redundant nodes from the quantized image processing model to obtain an initial image processing model; obtaining a target image processing model after deployment of the initial image processing model and an image to be processed; and performing analysis and processing on the image to be processed through the target image processing model to obtain image information.

Description

Technical Field

[0001] This invention relates to the field of neural networks, and more particularly to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] In the field of image recognition, the currently popular convolutional neural networks (also known as deep neural networks) are computationally intensive algorithms, requiring a large number of matrix multiplication operations. These neural networks are typically deployed on graphics processing units (GPUs) for computation. While this can reduce inference time to some extent, the overall computational load remains the same, and the memory consumption is still very large, especially considering that GPUs typically have limited memory. Therefore, reducing the computational load and memory usage of convolutional neural networks is a crucial topic in the field of deep learning.

[0003] Quantization is a common method to reduce the computational cost of a network. In the training and inference of deep neural networks, 32-bit floating-point numbers are typically used for computation. Quantization refers to the process of approximating a continuous 32-bit floating-point number as a finite set of discrete values. A common quantization method is to map the 32-bit floating-point number to an 8-bit integer, thus transforming floating-point calculations into integer (fixed-point) calculations.

[0004] The range of values ​​for a 32-bit floating-point number is much larger than that of an 8-bit integer. Therefore, directly mapping a 32-bit floating-point number symmetrically to an 8-bit integer would result in significant information loss. A common method to reduce information loss is post-training quantization. This involves inferring a certain size dataset onto the trained model to obtain the distribution of input (output) for each layer in the neural network. These distributions are then truncated using an interval-division algorithm and mapped to 8-bit values ​​to obtain another distribution. Methods such as KL divergence are then used to evaluate the approximation of the two distributions to determine an optimal truncation interval (also called a saturation interval). While this method can reduce the loss from model quantization and make the quantized distribution closer to the original model distribution, the accuracy may not meet ideal requirements in some practical applications. This is because the model weights do not actually change with quantization, meaning the model's robustness to quantized inputs may be poor. The problems of high computational cost, high memory usage, and low computational efficiency associated with image processing still exist. Summary of the Invention

[0005] This invention provides an image processing method, apparatus, device, and storage medium to reduce the computational load and memory consumption of image processing, and to lower its computational efficiency.

[0006] The first aspect of the present invention provides an image processing method, comprising:

[0007] The pre-trained image processing model is modified by inserting target quantization nodes.

[0008] The modified image processing model is subjected to quantization training to obtain a quantized image processing model;

[0009] Redundant nodes are removed from the quantized image processing model to obtain the initial image processing model.

[0010] The target image processing model after the initial image processing model is deployed, and the image to be processed are obtained. The image to be processed is then analyzed and processed by the target image processing model to obtain image information.

[0011] Optionally, in a first implementation of the first aspect of the present invention, the step of inserting target quantization nodes into the pre-trained image processing model to obtain a modified image processing model includes:

[0012] The pre-trained image processing model is used to identify target nodes to obtain nodes to be quantized, and the node type of the nodes to be quantized is obtained.

[0013] Based on the node type, a target quantization node is inserted into the node to be quantized to obtain a modified image processing model. The target quantization node is a node consisting of a quantization node followed by an inverse quantization node.

[0014] Optionally, in a second implementation of the first aspect of the present invention, the step of inserting a target quantization node into the node to be quantized based on the node type to obtain a modified image processing model includes:

[0015] When the node type is used to indicate a weighted node, the target quantization node is inserted per tensor and per channel to obtain the modified image processing model.

[0016] When the node type is used to indicate an additive node, target quantization nodes are inserted into the multiple inputs of the node to be quantized to obtain a modified image processing model.

[0017] Optionally, in a third implementation of the first aspect of the present invention, the step of quantizing the modified image processing model to obtain a quantized image processing model includes:

[0018] Obtain the result before quantization, the quantization ratio, and the zero point after quantization of the modified image processing model;

[0019] Based on the results before quantization, the quantization ratio, the zero point after quantization, and the preset quantization formula, the modified image processing model is calculated to obtain the quantized image processing model.

[0020] Optionally, in a fourth implementation of the first aspect of the present invention, the step of deleting redundant nodes from the quantized image processing model to obtain an initial image processing model includes:

[0021] Based on a preset graph optimization algorithm, multiple quantized training nodes in the quantized image processing model are obtained, and the continuity of the multiple quantized training nodes is identified to obtain continuity information.

[0022] Based on the aforementioned continuity information, redundant nodes in each quantization training node are identified to obtain the nodes to be deleted.

[0023] The nodes to be deleted from each quantized training node in the quantized image processing model are deleted to obtain the initial image processing model.

[0024] Optionally, in a fifth implementation of the first aspect of the present invention, the step of identifying redundant nodes in each quantized training node based on the continuity information to obtain the node to be deleted includes:

[0025] If the continuity information is used to indicate that two quantization training nodes are not continuous, then the inverse quantization node in the target quantization node of the quantization training node is obtained to obtain the first redundant node, and the first redundant node is determined as the node to be deleted.

[0026] If the continuity information is used to indicate that two quantization training nodes are continuous, then the target quantization node between the two quantization training nodes is obtained, and a second redundant node is obtained.

[0027] Obtain the dequantized node from the target quantized node of the quantized training node of the pre-connected quantized training node to obtain the third redundant node;

[0028] The second redundant node and the third redundant node are identified as nodes to be deleted.

[0029] Optionally, in a sixth implementation of the first aspect of the present invention, the step of obtaining the target image processing model after the initial image processing model is deployed, and the image to be processed, and analyzing and processing the image to be processed through the target image processing model to obtain image information, includes:

[0030] The initial image processing model is converted into a corresponding network structure representation to obtain the target image processing model, and the target image processing model is sent to the preset inference engine;

[0031] The system acquires the image to be processed and the image processing requirements, calls the target image processing model through the inference engine, and analyzes and processes the image to be processed based on the image processing requirements to obtain image information.

[0032] A second aspect of the present invention provides an image processing apparatus, comprising:

[0033] The insertion module is used to insert target quantization nodes into the pre-trained image processing model to obtain the modified image processing model.

[0034] The training module is used to perform quantization training on the modified image processing model to obtain the quantized image processing model.

[0035] The deletion module is used to remove redundant nodes from the quantized image processing model to obtain the initial image processing model.

[0036] The analysis and processing module is used to obtain the target image processing model after the initial image processing model is deployed, as well as the image to be processed, and to analyze and process the image to be processed through the target image processing model to obtain image information.

[0037] Optionally, in a first implementation of the second aspect of the present invention, the insertion module includes:

[0038] The recognition unit is used to identify target nodes in the pre-trained image processing model, obtain the nodes to be quantized, and acquire the node type of the nodes to be quantized.

[0039] An insertion unit is used to insert a target quantization node into the node to be quantized based on the node type, so as to obtain a modified image processing model. The target quantization node is a node consisting of a quantization node followed by an inverse quantization node.

[0040] Optionally, in a second implementation of the second aspect of the present invention, the insertion unit is specifically used for:

[0041] When the node type is used to indicate a weighted node, the target quantization node is inserted per tensor and per channel to obtain the modified image processing model.

[0042] When the node type is used to indicate an additive node, target quantization nodes are inserted into the multiple inputs of the node to be quantized to obtain a modified image processing model.

[0043] Optionally, in a third implementation of the second aspect of the present invention, the training module is specifically used for:

[0044] Obtain the result before quantization, the quantization ratio, and the zero point after quantization of the modified image processing model;

[0045] Based on the results before quantization, the quantization ratio, the zero point after quantization, and the preset quantization formula, the modified image processing model is calculated to obtain the quantized image processing model.

[0046] Optionally, in a fourth implementation of the second aspect of the present invention, the deletion module includes:

[0047] The first identification unit is used to obtain multiple quantized training nodes in the quantized image processing model based on a preset graph optimization algorithm, and to identify the continuity of the multiple quantized training nodes to obtain continuity information.

[0048] The second identification unit is used to identify redundant nodes in each quantized training node based on the continuity information, and to obtain the nodes to be deleted.

[0049] The deletion unit is used to delete the nodes to be deleted from each quantized training node in the quantized image processing model to obtain the initial image processing model.

[0050] Optionally, in a fifth implementation of the second aspect of the present invention, the second identification unit is specifically used for:

[0051] If the continuity information is used to indicate that two quantization training nodes are not continuous, then the inverse quantization node in the target quantization node of the quantization training node is obtained to obtain the first redundant node, and the first redundant node is determined as the node to be deleted.

[0052] If the continuity information is used to indicate that two quantization training nodes are continuous, then the target quantization node between the two quantization training nodes is obtained, and a second redundant node is obtained.

[0053] Obtain the dequantized node from the target quantized node of the quantized training node of the pre-connected quantized training node to obtain the third redundant node;

[0054] The second redundant node and the third redundant node are identified as nodes to be deleted.

[0055] Optionally, in a sixth implementation of the second aspect of the present invention, the analysis and processing module is specifically used for:

[0056] The initial image processing model is converted into a corresponding network structure representation to obtain the target image processing model, and the target image processing model is sent to the preset inference engine;

[0057] The system acquires the image to be processed and the image processing requirements, calls the target image processing model through the inference engine, and analyzes and processes the image to be processed based on the image processing requirements to obtain image information.

[0058] A third aspect of the present invention provides an image processing apparatus, comprising: a memory and at least one processor, wherein the memory stores a computer program; the at least one processor invokes the computer program in the memory to cause the image processing apparatus to perform the image processing method described above.

[0059] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the image processing method described above.

[0060] In the technical solution provided by this invention, a target quantization node is inserted into a pre-trained image processing model to obtain a modified image processing model; the modified image processing model is then subjected to quantization training to obtain a quantized image processing model; redundant nodes are removed from the quantized image processing model to obtain an initial image processing model; a target image processing model after the initial image processing model is deployed, along with the image to be processed, are obtained; and the image to be processed is analyzed and processed using the target image processing model to obtain image information. In this embodiment of the invention, by inserting a target quantization node (a combination of a quantization node and an inverse quantization node) before the node requiring quantization during deep neural network model training, the node is subjected to quantization interference during calculation, thus perceiving the existence of quantization. After training, the paired target quantization nodes are removed (i.e., redundant nodes are removed), and finally, the target image processing model is deployed to infer the image to be processed. This achieves compression of the image processing model, enabling the image processing model to benefit from quantization while avoiding the accuracy loss caused by quantization. This improves the robustness of the image processing model to quantized input, thereby reducing the computational load and memory consumption of image processing, and lowering its computational efficiency. Attached Figure Description

[0061] Figure 1 This is a schematic diagram of one embodiment of the image processing method in this invention;

[0062] Figure 2 This is a schematic diagram of another embodiment of the image processing method in this invention;

[0063] Figure 3 This is a schematic diagram of an embodiment of inserting a target quantization node into a node to be quantized according to the present invention;

[0064] Figure 4This is a schematic diagram of an embodiment of the present invention where the node type is used to indicate a weighted node for target quantization node insertion;

[0065] Figure 5 This is a schematic diagram of an embodiment of the present invention where the node type is used to indicate the insertion of the target quantization node;

[0066] Figure 6 This is a schematic diagram of an embodiment of the present invention in which the two quantization training nodes are not continuous;

[0067] Figure 7 This is a schematic diagram of an embodiment of the present invention in which two quantization training nodes are consecutive;

[0068] Figure 8 This is a schematic diagram of an embodiment of the present invention in which the node to be deleted in the quantization training nodes is deleted when the continuity information is used to indicate that two quantization training nodes are not continuous;

[0069] Figure 9 This is a schematic diagram of an embodiment of the present invention in which the node to be deleted in the quantization training nodes is deleted when the continuity information is used to indicate that two quantization training nodes are consecutive;

[0070] Figure 10 This is a schematic diagram of one embodiment of the image processing apparatus according to the present invention;

[0071] Figure 11 This is a schematic diagram of another embodiment of the image processing apparatus in this invention;

[0072] Figure 12 This is a schematic diagram of one embodiment of the image processing device in this invention. Detailed Implementation

[0073] The present invention provides an image processing method, apparatus, device, and storage medium, which reduces the computational load and memory consumption of image processing, and lowers its computational efficiency.

[0074] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0075] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the image processing method in this invention includes:

[0076] 101. Insert target quantization nodes into the pre-trained image processing model to obtain the modified image processing model.

[0077] It is understood that the executing entity of this invention can be an image processing device, a terminal, or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as an example.

[0078] The pre-trained image processing model is a deep neural network in the field of image recognition. Since quantization changes the features of the original model, if the model is trained with quantization from the beginning, its ability to learn some features will be reduced. Therefore, the model needs to be pre-trained in a non-quantized state. That is, the pre-trained image processing model is the image processing model pre-trained in a non-quantized state.

[0079] The server obtains the number of training epochs in the non-training mode of the original image processing model, trains the original image processing model based on the number of training epochs in the non-training mode, and determines the original image processing model that meets the training requirements after training as the pre-trained image processing model.

[0080] The server obtains the nodes to be quantized in the pre-trained image processing model, inserts the target quantization node at the target position of the node to be quantized, and modifies the model structure of the pre-trained image processing model to obtain the modified image processing model. The target position can be before and / or after, that is, the target position is used to indicate the before and / or after of the node to be quantized.

[0081] It should be noted that the target quantization node is used to indicate the quantization node that causes quantization interference to the node to be quantized in the pre-trained image processing model; the target quantization node includes quantization nodes and dequantization nodes; the number of inserted target quantization nodes can be one or more, which can be set according to the model quantization requirements, and is not limited here; the number of quantization nodes in the target quantization node can be one or more, which can be set according to the model quantization requirements, and is not limited here; the number of dequantization nodes in the target quantization node can be one or more, which can be set according to the model quantization requirements, and is not limited here.

[0082] 102. Perform quantization training on the modified image processing model to obtain the quantized image processing model.

[0083] After modifying the model structure of the pre-trained image processing model to obtain a modified image processing model, the server performs quantization training on the modified model. The server obtains the quantization training parameters and uses these parameters to perform quantization training on the modified image processing model, thereby obtaining a quantized image processing model. The quantization training parameters may include, but are not limited to, the result before quantization, the quantization ratio, and the zero point after quantization. The result before quantization indicates the output result of the pre-trained image processing model, such as prediction results or classification results, and is not limited here. It is set according to the output result (image processing function type) of the pre-trained image processing model. The zero point after quantization indicates the offset of a 32-bit floating-point number (i.e., a float32 value).

[0084] 103. Redundant nodes are removed from the quantized image processing model to obtain the initial image processing model.

[0085] The server uses a pre-defined redundant node detection strategy to identify redundant nodes in the quantized image processing model. Redundant nodes are used to indicate inserted inverse quantization nodes and / or target quantization nodes that affect the desired effect of the quantized image processing model. The redundant node detection strategy is used to detect redundant nodes; this strategy can be a graph optimization algorithm or pre-defined redundant node matching information, and is not limited here.

[0086] The server removes redundant nodes from the quantized image processing model to obtain the initial image processing model. In one feasible embodiment, after removing redundant nodes from the quantized image processing model, the server performs redundant node detection on the image processing model after redundant node removal to detect whether redundant nodes still exist. If redundant nodes exist, the corresponding redundant nodes are removed; otherwise, no processing is performed. This improves the accuracy and effectiveness of redundant node removal, thereby improving the effectiveness and robustness of the initial image processing model.

[0087] 104. Obtain the target image processing model after the initial image processing model is deployed, as well as the image to be processed, and analyze and process the image to be processed through the target image processing model to obtain image information.

[0088] After obtaining the initial image processing model, the server evaluates its accuracy and processing time. If the accuracy and processing time of the initial image processing model meet the preset conditions, it is designated as the target image processing model, deployed on the inference engine, and the image to be processed is acquired. The target image processing model is then used to analyze and process the image to obtain image information. If the accuracy and processing time of the initial image processing model do not meet the preset conditions, the server re-acquires the initial image processing model (i.e., it re-inserts target quantization nodes, performs quantization training, deletes redundant nodes, and evaluates accuracy and processing time on the pre-trained image processing model) until the accuracy and processing time of the re-acquired initial image processing model meet the preset conditions. The re-acquired initial image processing model is then designated as the target image processing model, deployed on the inference engine, and the image to be processed is acquired. The target image processing model is then used to analyze and process the image to obtain image information.

[0089] As an example, and not a limitation, the target image processing model is deployed on an autonomous vehicle, and images are acquired during autonomous driving through at least one camera on the autonomous vehicle to obtain the image to be processed; the analysis and processing of the image to be processed by the target image processing model can be image classification, image detection, or image segmentation, etc.

[0090] In this embodiment of the invention, during the training of the deep neural network model, a target quantization node (a combination of a quantization node and an anti-quantization node) is inserted before the node that needs to be quantized. This causes the node to be disturbed by quantization during calculation, thus perceiving the existence of quantization. After training is completed, the paired target quantization nodes are removed (i.e., redundant nodes are deleted). Finally, the target image processing model is deployed to perform inference on the image to be processed. This achieves compression of the image processing model, enabling the image processing model to benefit from quantization while avoiding the accuracy loss caused by quantization. It also improves the robustness of the image processing model to input quantization, thereby reducing the computational load and memory consumption of image processing, and lowering its computational efficiency.

[0091] Please see Figure 2 Another embodiment of the image processing method in this invention includes:

[0092] 201. Perform target node identification on the pre-trained image processing model to obtain the nodes to be quantized and obtain the node type of the nodes to be quantized.

[0093] The target node is used to indicate the node to be quantized. The target node may include, but is not limited to, weighted nodes and additive nodes. Weighted nodes include, for example, common weighted nodes such as convolution and fully connected nodes.

[0094] The server performs target node identification and type labeling on the pre-trained image processing model to obtain node information, which includes the node to be quantized and its type labeling information. The type labeling information of the node to be quantized and its type are extracted from the node information to obtain the node to be quantized and its type.

[0095] It should be noted that for more critical nodes, such as those used in regression, if we want to maintain high accuracy, we can leave them unprocessed. That is, the pre-defined critical nodes are not used as nodes to be quantized in order to maintain high accuracy.

[0096] 202. Based on the node type, insert the target quantization node into the node to be quantized to obtain the modified image processing model. The target quantization node is a node consisting of a quantization node followed by an inverse quantization node.

[0097] In this context, a target quantization node is a node consisting of a quantization node followed immediately by a dequantization node. The quantization node is used to quantize the high-bit value to the low-bit value, and the dequantization node is used to dequantize the low-bit value back to the high-bit value. This is an example, not a limitation, of... Figure 3 As shown, Figure 3The diagram illustrates an embodiment of inserting a target quantization node into a node to be quantized. Q is the quantization node, DQ is the dequantization node, QDQ is the pseudo-quantization node (Q is immediately followed by DQ), and OP is the node to be quantized.

[0098] Specifically, when the node type is used to indicate a weighted node, the server performs tensor-by-tensor target quantization node insertion and channel-by-channel target quantization node insertion on the node to be quantized to obtain the modified image processing model; when the node type is used to indicate an additive node, target quantization node insertion is performed on multiple inputs of the node to be quantized to obtain the modified image processing model.

[0099] The target node may include, but is not limited to, weighted nodes and additive nodes. As an example and not a limitation, this embodiment uses node types indicating weighted nodes and node types indicating additive nodes for illustration. Figure 4 As shown, Figure 4 This diagram illustrates an embodiment of target quantization node insertion when the node type is used to indicate a weighted node. The weighted node is a convolutional node. A target quantization node is inserted between the convolutional node and the input to achieve per-tensor target quantization node insertion, thereby quantizing the entire feature of the input to the node to be quantized. A target quantization node is also inserted between the convolutional node and the weights to achieve per-channel target quantization node insertion. Figure 5 As shown, Figure 5 The diagram illustrates an embodiment of target quantization node insertion when the node type is used to indicate an adder node. The adder node corresponds to two inputs, input A and input B. A target quantization node is inserted between the adder node and input A, and another target quantization node is inserted between the adder node and input B to ensure that both sides of the adder node are quantized to low bits, that is, to ensure that all sides of the node to be quantized are quantized to low bits.

[0100] This method enables the insertion of target quantization nodes based on the different characteristics of the nodes to be quantized, thereby improving the accuracy and effectiveness of target quantization node insertion. This leads to the compression of the image processing model, allowing the image processing model to benefit from quantization while avoiding the accuracy loss caused by quantization, and improving the robustness of the image processing model to the quantization of inputs.

[0101] 203. Perform quantization training on the modified image processing model to obtain the quantized image processing model.

[0102] Specifically, the server obtains the pre-quantization result, quantization ratio, and post-quantization zero point of the modified image processing model; based on the pre-quantization result, quantization ratio, post-quantization zero point, and preset quantization formula, it performs calculations on the modified image processing model to obtain the quantized image processing model.

[0103] The result before quantization indicates the output of the pre-trained image processing model, such as prediction or classification results. This is not limited to specific values ​​and is set based on the output of the pre-trained image processing model (image processing function type). The zero point after quantization indicates the offset of a 32-bit floating-point number (i.e., a float32 value). The server obtains the maximum and minimum values ​​of the input to the modified image processing model and calculates the quantization ratio based on these values.

[0104] For example, based on the result before quantization, the quantization ratio, the zero point after quantization, and the preset quantization formula, the modified image processing model is calculated to obtain the quantized result, i.e., the quantized image processing model. The preset quantization formula is as follows: Q = round(R / S) + Z, where Q represents the quantized result, i.e., the quantized image processing model, round represents the integer rounding operation, R represents the result before quantization, S represents the quantization ratio, and Z represents the zero point after quantization. When calculating the gradient of the modified image processing model for quantization training using the result before quantization, the quantization ratio, the zero point after quantization, and the above preset quantization formula, the gradient of the quantization training is obtained by passing through the gradient of the nodes propagated from the backpropagation. That is, the gradient is passed through by directly propagating the gradient during the backpropagation process, skipping the calculation of the gradient of the quantization nodes.

[0105] By quantizing the modified image processing model, the model can benefit from quantization while avoiding the loss of accuracy caused by quantization, thus improving the robustness of the image processing model to quantized inputs.

[0106] 204. Redundant nodes are removed from the quantized image processing model to obtain the initial image processing model.

[0107] Specifically, the server uses a pre-set graph optimization algorithm to obtain multiple quantized training nodes in the quantized image processing model, and identifies the continuity of the multiple quantized training nodes to obtain continuity information; based on the continuity information, it identifies redundant nodes in each quantized training node to obtain nodes to be deleted; and deletes the nodes to be deleted in each quantized training node in the quantized image processing model to obtain the initial image processing model.

[0108] In this system, quantization training nodes indicate the nodes formed after inserting a target quantization node into a node to be quantized. Continuity information indicates whether two quantization training nodes are consecutive (whether there is continuity between the two quantization training nodes), and the continuity of quantization training nodes indicates whether the two quantization training nodes are tightly connected. The server identifies redundant nodes in each quantization training node based on whether two quantization training nodes are consecutive (whether the two quantization training nodes are tightly connected), thereby identifying inserted inverse quantization nodes and / or target quantization nodes that affect the expected effect of the quantized image processing model—i.e., nodes to be deleted. The server then deletes these nodes from each quantization training node in the quantized image processing model, obtaining the initial image processing model. This improves the accuracy and effectiveness of redundant node deletion, thus enhancing the effectiveness and robustness of the initial image processing model.

[0109] Specifically, the server identifies redundant nodes in each quantization training node based on continuity information to obtain nodes to be deleted, including: if the continuity information indicates that two quantization training nodes are not continuous, then the server obtains the inverse quantization node in the target quantization node of the quantization training node to obtain a first redundant node, and determines the first redundant node as a node to be deleted; if the continuity information indicates that two quantization training nodes are continuous, then the server obtains the target quantization node between the two quantization training nodes to obtain a second redundant node; the server obtains the inverse quantization node in the target quantization node of the previously connected quantization training node to obtain a third redundant node; and the server determines the second and third redundant nodes as nodes to be deleted.

[0110] The discontinuity of the two quantization training nodes can mean that the two quantization training nodes are not closely connected, or that the two quantization training nodes are each placed separately in their respective structural positions. This is an example, not a limitation, of... Figure 6 As shown, the two quantization training nodes are not contiguous. Q1, DQ1, and OP1 constitute the first quantization training node, and Q2, DQ2, and OP2 constitute the second quantization training node. The first and second quantization training nodes are not tightly connected, or the first (or second) quantization training node is a node placed independently at each structural position. Two quantization training nodes being contiguous means that the two quantization training nodes are tightly connected. This is an example, not a limitation, such as... Figure 7 As shown, the two quantization training nodes are consecutive, Q3, DQ3 and OP3 constitute the third quantization training node, and Q4, DQ4 and OP4 constitute the fourth quantization training node. The third quantization training node and the fourth quantization training node are closely connected.

[0111] For example, if continuity information is used to indicate that two quantized training nodes are not continuous (e.g.) Figure 6The diagram shows a first quantization training node composed of Q1, DQ1, and OP1, and a second quantization training node composed of Q2, DQ2, and OP2. Since DQ (the dequantization node) dequantizes low-bit values ​​back to high-bit values, which is not the desired quantization effect, the dequantization node DQ1 in the target quantization node of the first quantization training node is obtained, resulting in the first redundant node 1. Similarly, the dequantization node DQ2 in the target quantization node of the second quantization training node is obtained, resulting in the first redundant node 2. The first redundant node 1 and the first redundant node 2 are then designated as node 1 to be deleted and node 2 to be deleted, respectively. The nodes after deleting node 1 and node 2 are shown below. Figure 8 As shown; if continuity information is used to indicate that two quantized training nodes are continuous (e.g. Figure 7 The diagram shows the third quantization training node (composed of Q3, DQ3, and OP3) and the fourth quantization training node (composed of Q4, DQ4, and OP4). Since both the third and fourth quantization training nodes have undergone quantization training, OP2 also expects to perform computation at low bit depths during deployment. Therefore, the dequantization node QDQ (Q4DQ4) of OP2 is redundant. The target quantization node between the two quantization training nodes is obtained, resulting in the second redundant node, i.e., the dequantization node QDQ (Q4DQ4) of OP2. The dequantization node in the target quantization node QDQ (Q3DQ3) of the previously connected quantization training node (i.e., the third quantization training node) is obtained, resulting in the third redundant node, DQ3. The second redundant node Q4DQ4 and the third redundant node DQ3 are designated as node 3 and node 4 to be deleted, respectively. The nodes after deleting node 3 and node 4 are as follows: Figure 9 As shown.

[0112] By identifying redundant nodes in each quantization training node based on continuous information, the accuracy and effectiveness of redundant nodes are improved.

[0113] 205. Obtain the target image processing model after the initial image processing model is deployed, as well as the image to be processed, and analyze and process the image to be processed through the target image processing model to obtain image information.

[0114] Specifically, the server converts the initial image processing model into a corresponding network structure representation to obtain the target image processing model, and sends the target image processing model to the pre-set inference engine; it obtains the image to be processed and the image processing requirements, calls the target image processing model through the inference engine, and analyzes and processes the image to be processed based on the image processing requirements to obtain image information.

[0115] The server converts the initial image processing model into a general network structure representation such as ONNX to obtain the target image processing model. This target model is then provided to a pre-built inference engine for deployment, such as TensorRT for low-bit computation. In autonomous driving scenarios, convolutional neural networks for image classification, detection, and segmentation all have a certain quantization space. Specifically, the inference process (analysis) of the quantized model (i.e., the target image processing model) is the same as that of the non-quantized network (i.e., the pre-trained image processing model), except that the intermediate features of the network are low-precision, and the operators or hardware involved in the computation are optimized for low precision.

[0116] After obtaining the initial image processing model, the server evaluates its accuracy and processing time. If the accuracy and processing time of the initial image processing model meet the preset conditions, it is designated as the target image processing model, deployed on the inference engine, and the image to be processed is acquired. The target image processing model is then used to analyze and process the image to obtain image information. If the accuracy and processing time of the initial image processing model do not meet the preset conditions, the server re-acquires the initial image processing model (i.e., it re-inserts target quantization nodes, performs quantization training, deletes redundant nodes, and evaluates accuracy and processing time on the pre-trained image processing model) until the accuracy and processing time of the re-acquired initial image processing model meet the preset conditions. The re-acquired initial image processing model is then designated as the target image processing model, deployed on the inference engine, and the image to be processed is acquired. The target image processing model is then used to analyze and process the image to obtain image information.

[0117] By deploying the target image processing model onto the inference engine to analyze and process the image to be processed, the computational load and memory consumption of image processing are reduced, as well as its computational efficiency is lowered.

[0118] In this embodiment of the invention, during the training of the deep neural network model, a target quantization node (a combination of a quantization node and an anti-quantization node) is inserted before the node that needs to be quantized. This causes the node to be disturbed by quantization during calculation, thus perceiving the existence of quantization. After training, the paired target quantization nodes are eliminated (i.e., redundant nodes are deleted) using a graph optimization algorithm. Finally, the target image processing model is deployed to perform inference on the image to be processed. This achieves compression of the image processing model, enabling the image processing model to benefit from quantization while avoiding the accuracy loss caused by quantization. It also improves the robustness of the image processing model to input quantization, thereby reducing the computational load and memory consumption of image processing, and lowering its computational efficiency.

[0119] The image processing method in the embodiments of the present invention has been described above. The image processing apparatus in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 10 One embodiment of the image processing apparatus in this invention includes:

[0120] The insertion module 1010 is used to insert target quantization nodes into the pre-trained image processing model to obtain the modified image processing model.

[0121] Training module 1020 is used to perform quantization training on the modified image processing model to obtain the quantized image processing model;

[0122] The deletion module 1030 is used to delete redundant nodes in the quantized image processing model to obtain the initial image processing model.

[0123] The analysis and processing module 1040 is used to obtain the target image processing model after the initial image processing model is deployed, as well as the image to be processed, and to analyze and process the image to be processed through the target image processing model to obtain image information.

[0124] The functions of each module in the above-mentioned image processing device correspond to the steps in the above-mentioned image processing method embodiment, and their functions and implementation processes will not be described in detail here.

[0125] In this embodiment of the invention, during the training of the deep neural network model, a target quantization node (a combination of a quantization node and an anti-quantization node) is inserted before the node that needs to be quantized. This causes the node to be disturbed by quantization during calculation, thus perceiving the existence of quantization. After training is completed, the paired target quantization nodes are removed (i.e., redundant nodes are deleted). Finally, the target image processing model is deployed to perform inference on the image to be processed. This achieves compression of the image processing model, enabling the image processing model to benefit from quantization while avoiding the accuracy loss caused by quantization. It also improves the robustness of the image processing model to input quantization, thereby reducing the computational load and memory consumption of image processing, and lowering its computational efficiency.

[0126] Please see Figure 11 Another embodiment of the image processing apparatus in this invention includes:

[0127] The insertion module 1010 is used to insert target quantization nodes into the pre-trained image processing model to obtain the modified image processing model.

[0128] Specifically, the insertion module 1010 includes:

[0129] The recognition unit 1011 is used to identify target nodes in the pre-trained image processing model, obtain the nodes to be quantized, and acquire the node type of the nodes to be quantized.

[0130] The insertion unit 1012 is used to insert a target quantization node into the node to be quantized based on the node type, so as to obtain a modified image processing model. The target quantization node is a node consisting of a quantization node followed by an inverse quantization node.

[0131] Training module 1020 is used to perform quantization training on the modified image processing model to obtain the quantized image processing model;

[0132] The deletion module 1030 is used to delete redundant nodes in the quantized image processing model to obtain the initial image processing model.

[0133] The analysis and processing module 1040 is used to obtain the target image processing model after the initial image processing model is deployed, as well as the image to be processed, and to analyze and process the image to be processed through the target image processing model to obtain image information.

[0134] Optionally, the insertion unit 1012 can also be specifically used for:

[0135] When the node type is used to indicate a weighted node, the target quantization node is inserted per tensor and per channel to obtain the modified image processing model.

[0136] When the node type is used to indicate an additive node, the target quantization node is inserted into the multiple inputs of the node to be quantized, resulting in a modified image processing model.

[0137] Optionally, the training module 1020 can also be specifically used for:

[0138] Obtain the result before quantization, the quantization ratio, and the zero point after quantization of the modified image processing model;

[0139] Based on the results before quantization, the quantization ratio, the zero point after quantization, and the preset quantization formula, the modified image processing model is calculated to obtain the quantized image processing model.

[0140] Optionally, the deletion module 1030 includes:

[0141] The first recognition unit 1031 is used to obtain multiple quantized training nodes in the quantized image processing model based on a preset graph optimization algorithm, and to identify the continuity of the multiple quantized training nodes to obtain continuity information.

[0142] The second identification unit 1032 is used to identify redundant nodes in each quantized training node based on continuous information to obtain the nodes to be deleted.

[0143] The deletion unit 1033 is used to delete the nodes to be deleted from each quantized training node in the quantized image processing model to obtain the initial image processing model.

[0144] Optionally, the second identification unit 1032 can also be specifically used for:

[0145] If the continuity information is used to indicate that two quantization training nodes are not continuous, then the inverse quantization node in the target quantization node of the quantization training node is obtained to obtain the first redundant node, and the first redundant node is determined as the node to be deleted.

[0146] If the continuity information is used to indicate that two quantization training nodes are continuous, then the target quantization node between the two quantization training nodes is obtained, and the second redundant node is obtained.

[0147] Obtain the dequantized node from the target quantized node of the quantized training node of the pre-connected quantized training node to obtain the third redundant node;

[0148] The second and third redundant nodes are identified as nodes to be deleted.

[0149] Optionally, the analysis and processing module 1040 can also be specifically used for:

[0150] The initial image processing model is converted into the corresponding network structure expression to obtain the target image processing model, and the target image processing model is sent to the pre-built inference engine;

[0151] The system acquires the image to be processed and the image processing requirements, calls the target image processing model through the inference engine, and analyzes and processes the image to be processed based on the image processing requirements to obtain image information.

[0152] The functions of each module and unit in the above-mentioned image processing device correspond to the steps in the above-mentioned image processing method embodiment, and their functions and implementation processes will not be described in detail here.

[0153] In this embodiment of the invention, during the training of the deep neural network model, a target quantization node (a combination of a quantization node and an anti-quantization node) is inserted before the node that needs to be quantized. This causes the node to be disturbed by quantization during calculation, thus perceiving the existence of quantization. After training, the paired target quantization nodes are eliminated (i.e., redundant nodes are deleted) using a graph optimization algorithm. Finally, the target image processing model is deployed to perform inference on the image to be processed. This achieves compression of the image processing model, enabling the image processing model to benefit from quantization while avoiding the accuracy loss caused by quantization. It also improves the robustness of the image processing model to input quantization, thereby reducing the computational load and memory consumption of image processing, and lowering its computational efficiency.

[0154] above Figure 10 and Figure 11 The image processing apparatus in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The image processing device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0155] Figure 12 This is a schematic diagram of an image processing device according to an embodiment of the present invention. The image processing device 1200 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 1210 (e.g., one or more processors) and a memory 1220, and one or more storage media 1230 (e.g., one or more mass storage devices) for storing application programs 1233 or data 1232. The memory 1220 and storage media 1230 can be temporary or persistent storage. The program stored in the storage media 1230 may include one or more modules (not shown in the diagram), each module including a series of computer program operations on the image processing device 1200. Furthermore, the processor 1210 may be configured to communicate with the storage media 1230 and execute the series of computer program operations in the storage media 1230 on the image processing device 1200.

[0156] The image processing device 1200 may also include one or more power supplies 1240, one or more wired or wireless network interfaces 1250, one or more input / output interfaces 1260, and / or one or more operating systems 1231, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 12 The illustrated image processing device structure does not constitute a limitation on the image processing device and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0157] The present invention also provides an image processing apparatus, comprising: a memory and at least one processor, wherein the memory stores a computer program, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the computer program in the memory to cause the image processing apparatus to perform the steps in the image processing method described above.

[0158] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein a computer program is stored in the computer program, which, when run on a computer, causes the computer to perform the steps of an image processing method.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0160] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several computer programs to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method, characterized in that, The image processing method includes: A modified image processing model is obtained by inserting target quantization nodes into the pre-trained image processing model; the target quantization nodes are used to indicate quantization nodes that cause quantization interference to the nodes to be quantized in the pre-trained image processing model; the target quantization nodes include quantization nodes and dequantization nodes. The modified image processing model is subjected to quantization training to obtain a quantized image processing model; Redundant nodes are removed from the quantized image processing model to obtain the initial image processing model. The target image processing model after the initial image processing model is deployed, and the image to be processed are obtained. The image to be processed is analyzed and processed by the target image processing model to obtain image information. The step of inserting target quantization nodes into the pre-trained image processing model to obtain the modified image processing model includes: The pre-trained image processing model is used to identify target nodes to obtain nodes to be quantized, and the node type of the nodes to be quantized is obtained. Based on the node type, a target quantization node is inserted into the node to be quantized to obtain a modified image processing model. The target quantization node is a node consisting of a quantization node followed by an inverse quantization node. The step of inserting a target quantization node into the node to be quantized based on the node type to obtain a modified image processing model includes: When the node type is used to indicate a weighted node, the target quantization node is inserted per tensor and per channel to obtain the modified image processing model. When the node type is used to indicate an addition node, target quantization nodes are inserted into the multiple inputs of the node to be quantized to obtain a modified image processing model. The process of removing redundant nodes from the quantized image processing model to obtain an initial image processing model includes: Based on a preset graph optimization algorithm, multiple quantized training nodes in the quantized image processing model are obtained, and the continuity of the multiple quantized training nodes is identified to obtain continuity information. Based on the aforementioned continuity information, redundant nodes in each quantization training node are identified to obtain the nodes to be deleted. The nodes to be deleted in each quantized training node of the quantized image processing model are deleted to obtain the initial image processing model. The step of identifying redundant nodes in each quantized training node based on the continuity information to obtain nodes to be deleted includes: If the continuity information is used to indicate that two quantization training nodes are not continuous, then the inverse quantization node in the target quantization node of the quantization training node is obtained to obtain the first redundant node, and the first redundant node is determined as the node to be deleted. If the continuity information is used to indicate that two quantization training nodes are continuous, then the target quantization node between the two quantization training nodes is obtained, and a second redundant node is obtained. Obtain the dequantized node from the target quantized node of the quantized training node of the pre-connected quantized training node to obtain the third redundant node; The second redundant node and the third redundant node are identified as nodes to be deleted.

2. The image processing method according to claim 1, characterized in that, The step of quantizing and training the modified image processing model to obtain a quantized image processing model includes: Obtain the result before quantization, the quantization ratio, and the zero point after quantization of the modified image processing model; Based on the results before quantization, the quantization ratio, the zero point after quantization, and the preset quantization formula, the modified image processing model is calculated to obtain the quantized image processing model.

3. The image processing method according to any one of claims 1-2, characterized in that, The process of acquiring the target image processing model after the initial image processing model is deployed, and the image to be processed, and then analyzing and processing the image to be processed using the target image processing model to obtain image information, includes: The initial image processing model is converted into a corresponding network structure representation to obtain the target image processing model, and the target image processing model is sent to the preset inference engine; The system acquires the image to be processed and the image processing requirements, calls the target image processing model through the inference engine, and analyzes and processes the image to be processed based on the image processing requirements to obtain image information.

4. An image processing apparatus, characterized in that, The image processing apparatus includes: An insertion module is used to insert target quantization nodes into a pre-trained image processing model to obtain a modified image processing model; the target quantization node is used to indicate the quantization node that causes quantization interference to the node to be quantized in the pre-trained image processing model; the target quantization node includes quantization nodes and dequantization nodes; The training module is used to perform quantization training on the modified image processing model to obtain a quantized image processing model. The deletion module is used to delete redundant nodes in the quantized image processing model to obtain an initial image processing model. The analysis and processing module is used to obtain the target image processing model after the initial image processing model is deployed, as well as the image to be processed, and to analyze and process the image to be processed through the target image processing model to obtain image information; The insertion module is also used to identify target nodes in the pre-trained image processing model, obtain the nodes to be quantized, and acquire the node type of the nodes to be quantized. Based on the node type, a target quantization node is inserted into the node to be quantized to obtain a modified image processing model. The target quantization node is a node consisting of a quantization node followed by an inverse quantization node. The insertion module is also used to insert target quantization nodes per tensor and per channel when the node type is used to indicate a weighted node, so as to obtain a modified image processing model. When the node type is used to indicate an addition node, target quantization nodes are inserted into the multiple inputs of the node to be quantized to obtain a modified image processing model. The deletion module is also used to obtain multiple quantized training nodes in the quantized image processing model based on a preset graph optimization algorithm, and to identify the continuity of the multiple quantized training nodes to obtain continuity information. Based on the aforementioned continuity information, redundant nodes in each quantization training node are identified to obtain the nodes to be deleted. The nodes to be deleted in each quantized training node of the quantized image processing model are deleted to obtain the initial image processing model. The deletion module is further configured to, if the continuity information is used to indicate that two quantization training nodes are not continuous, obtain the inverse quantization node in the target quantization node of the quantization training node, obtain the first redundant node, and determine the first redundant node as the node to be deleted. If the continuity information is used to indicate that two quantization training nodes are continuous, then the target quantization node between the two quantization training nodes is obtained, and a second redundant node is obtained. Obtain the dequantized node from the target quantized node of the quantized training node of the pre-connected quantized training node to obtain the third redundant node; The second redundant node and the third redundant node are identified as nodes to be deleted.

5. An image processing apparatus, characterized in that, The image processing device includes: a memory and at least one processor, wherein the memory stores a computer program; The at least one processor invokes the computer program in the memory to cause the image processing device to perform the image processing method as described in any one of claims 1-3.

6. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1-3.

Images