Image processing method and storage device
By fusing operators in the image processing model to generate a fused model and calculating similarity, the problem of slow inference speed of the image processing model on the camera is solved, and the accuracy of the inference results is maintained while simplifying the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2022-12-29
- Publication Date
- 2026-05-29
AI Technical Summary
When image processing models are applied to cameras, the large size of the model leads to slow inference speed, and it is difficult to guarantee the accuracy of the inference results.
By fusing adjacent operators in the initial image processing model, a fused image processing model is generated. The initial and fused models are then used to infer the same image data, and the similarity is calculated to determine the final output fused model, ensuring that the accuracy of the inference results is maintained while the model is simplified.
While improving inference speed, it ensures the accuracy of inference results of the image processing model, making it suitable for image processing tasks using cameras.
Smart Images
Figure CN116228513B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of artificial intelligence technology, specifically relating to an image processing method and a storage device. Background Technology
[0002] Image processing models have shown good performance in various computer vision tasks; however, they often have many layers and a large number of parameters. For example, the AlexNet model with only 8 layers has about 60 million parameters and requires about 700 million floating-point calculations. Therefore, the application of image processing models in real-world scenarios has many limitations; for example, when applying image processing models to cameras, the large size of the models results in slow inference speed. Summary of the Invention
[0003] This application provides an image processing method and storage device to reduce the size of the image processing model applied to the camera, thereby improving inference speed while ensuring the accuracy of inference results.
[0004] To address the aforementioned technical problems, this application provides an image processing method comprising: a camera acquiring a first fused image processing model output by a processor; wherein the process of the processor outputting the first fused image processing model includes: fusing at least two adjacent operators in an initial image processing model to obtain a fused first fused image processing model; obtaining a first similarity between a first inference result and a second inference result; wherein the first inference result is obtained by inferring from first image data using the initial image processing model, and the second inference result is obtained by inferring from the first image data using the first fused image processing model; outputting the first fused image processing model in response to the first similarity being greater than or equal to a threshold; and using the first fused image processing model to infer from video frames to obtain corresponding inference results.
[0005] To address the aforementioned technical problems, another technical solution adopted in this application is: providing an image processing method, comprising: a processor fusing at least two adjacent operators in an initial image processing model to obtain a fused first image processing model; obtaining a first similarity between a first inference result and a second inference result; wherein the first inference result is obtained by the initial image processing model inferring from the first image data, and the second inference result is obtained by the first fused image processing model inferring from the first image data; in response to the first similarity being greater than or equal to a threshold, outputting the first fused image processing model so that a camera uses the first fused image processing model to infer from video frames to obtain corresponding inference results.
[0006] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a storage device that stores program instructions that can be executed by a processor, the program instructions being used to implement the image processing method described in any of the above embodiments.
[0007] Unlike existing technologies, the beneficial effects of this application are as follows: In the image processing method provided by this application, after the processor performs operator fusion on the initial image processing model to obtain a fused first image processing model, it uses the initial image processing model and the first fused image processing model to perform inference on the same first image data to obtain a first inference result and a second inference result, respectively. Subsequently, the first fused image processing model is tested based on the first similarity between the first inference result and the second inference result, so that the final output first fused image processing model is simplified while the accuracy of the inference result is high. When this first fused image processing model is applied to a camera, it can improve the inference speed while ensuring the accuracy of the inference result. Attached Figure Description
[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:
[0009] Figure 1 This is a flowchart illustrating one embodiment of the image processing method of this application;
[0010] Figure 2 for Figure 1 A flowchart of one embodiment corresponding to step S101;
[0011] Figure 3 This is a connection diagram of one implementation of a partial operator;
[0012] Figure 4 for Figure 1 A flowchart illustrating another implementation method following step S102;
[0013] Figure 5 This is a flowchart illustrating another embodiment of the image processing method of this application.
[0014] Figure 6 This is a schematic diagram of the structure of one embodiment of the image processing apparatus of this application;
[0015] Figure 7 This is a schematic diagram of the structure of one embodiment of the electronic device of this application;
[0016] Figure 8 This is a schematic diagram of the structure of one embodiment of the storage device of this application. Detailed Implementation
[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0018] To facilitate understanding, the image processing method provided in this application will first be introduced from the processor's perspective. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the image processing method of this application. The image processing method includes:
[0019] S101: Fuse at least two adjacent operators in the initial image processing model to obtain the first fused image processing model.
[0020] Specifically, the initial image processing model mentioned above can be an object detection model, an object tracking module, or an image classification model, etc.; and the framework of the initial image processing model can be any one of PyTorch, TensorFlow, ONNX, and Caffe. In some scenarios, layers with the same function may have different names in different frameworks. The specific operator fusion methods mentioned later in this application can apply layers with the same function in different frameworks.
[0021] In one implementation, please refer to Figure 2 , Figure 2 for Figure 1 A flowchart illustrating one embodiment corresponding to step S101. Step S101 specifically includes:
[0022] S201: Obtain one operator sequentially starting from the input of the initial image processing model.
[0023] Specifically, since operator fusion in this application mainly involves fusing two adjacent operators, a depth-first traversal can be performed on the initial image processing model.
[0024] S202: Determine whether the current operator is a preset operator and whether the number of branches connected to the output of the current operator is one; if yes, proceed to step S203; if no, proceed to step S205.
[0025] Optionally, in this embodiment, there can be multiple preset operators; for example, the preset operators include the convolution operator Conv and the shape operator Shape; the above determination of whether the current operator is a preset operator is equivalent to determining whether the current operator is the convolution operator Conv or the shape operator Shape. Of course, in other embodiments, there can be only one preset operator; for example, the preset operator is only the convolution operator Conv or the shape operator Shape.
[0026] Furthermore, the purpose of operator fusion is to simplify the model and accelerate inference, but this requires that the models before and after fusion be equivalent. For some complex models, fusion of certain layers may lead to errors. For example, ... Figure 3 As shown, Figure 3 This is a connection diagram for one implementation of a partial operator. Figure 3 The output of convolutional layer 1 is connected to two branches: a normalization layer and convolutional layer 2. If convolutional layer 1 and the normalization layer are fused, the branch containing convolutional layer 2 will have an additional normalization layer calculation, which may lead to inconsistencies in the inference results before and after fusion. Therefore, the judgment condition in step S202 includes a step of determining whether the number of branches connected to the output of the current operator is one.
[0027] Furthermore, the meaning of "if" above is: if the current operator is a preset operator, and the number of branches connected to the output of the current operator is one; the meaning of "if not" above is: if the current operator is not a preset operator, or if the current operator is a preset operator, but the number of branches connected to the output of the current operator is greater than or equal to two.
[0028] S203: Determine whether the current operator and another operator connected to the output of the current operator conform to the preset fusion strategy; if yes, proceed to step S204; if no, proceed to step S205.
[0029] Specifically, when there are multiple preset operators, the preset fusion strategies corresponding to different preset operators are different, and each preset fusion strategy corresponding to a preset operator contains multiple combinations of fusionable operators. For example, the preset operators include the convolution operator Conv and the shape operator Shape, and their corresponding preset fusion strategies are divided into two main categories: the Convolution operator Conv + other operator types fusion method and the Shape operator Shape + other operator types fusion method.
[0030] In detail, when the preset operator is the convolution operator Conv, the combination of fusionable operators included in the corresponding preset fusion strategy may include: convolution operator Conv + normalization operator Bn, convolution operator Conv + activation operator Activation, convolution operator Conv + computation function operator (e.g., addition operator Add, subtraction operator Sub, multiplication operator Mul, division operator Div).
[0031] When the preset operator is the Shape operator, the combination of fusionable operators included in its corresponding preset fusion strategy includes: Shape operator + extraction function operator (e.g., Gather operator, Slice operator), Shape operator + concatenation function operator, Shape operator + calculation function operator (e.g., Add operator, Sub operator, Mul operator, Div operator).
[0032] In one implementation, step S203 can be implemented as follows: obtaining the preset fusion strategy corresponding to the current operator, and determining whether the current operator and another operator connected to the output of the current operator are among the fusionable operator combinations of the corresponding preset fusion strategy. Further, "if" means that the combination of the current operator and the other operator is one of the fusionable operator combinations of the corresponding preset fusion strategy; "if not" means that the combination of the current operator and the other operator is not one of the fusionable operator combinations of the corresponding preset fusion strategy.
[0033] In another implementation, in certain situations, when the current operator is one of the preset operators, and multiple other operators that can be fused with the current operator are sequentially connected to the same branch after it, in order to reduce the probability of errors occurring during multiple fusion operations, different combinations of fused operators in the preset fusion strategy corresponding to the same preset operator have different priorities, and the specific priorities can be set according to actual experience.
[0034] For example, when the preset operator is the convolution operator Conv, the priorities of the convolution operator Conv + normalization operator Bn, the convolution operator Conv + activation operator Activation, and the convolution operator Conv + computation function operator decrease in that order; that is, Conv + Bn has the highest priority, Conv + Activation has the second highest priority, and Conv + computation function operator has the lowest priority.
[0035] When the preset operator is Shape, the priority of Shape + Extraction Operator, Shape + Concat Operator, and Shape + Calculation Operator decreases in that order; that is, Shape + Extraction Operator has the highest priority, Shape + Concat has the second highest priority, and Shape + Calculation Operator has the lowest priority.
[0036] Furthermore, the specific implementation process of step S203 above can be as follows: obtain the first priority of the fusionable operator combination corresponding to the current operator and another operator connected to its output; determine whether the first priority is the highest priority, or the first priority is not the highest priority, and whether the current operator is formed by fusing a combination of fusionable operators with a priority greater than or equal to the first priority. In this case, "if" means that the first priority is the highest priority under the corresponding preset fusion strategy; or, the first priority is not the highest priority, and the current operator is formed by fusing a combination of fusionable operators with a priority greater than or equal to the first priority. "If not" means that the first priority is not the highest priority, and the current operator is not formed by fusing a combination of fusionable operators with a priority greater than or equal to the first priority.
[0037] For example, some common model structures in practice are Conv+Bn+Activation, where the output of Conv is connected to Bn, and the output of Bn is connected to Activation. If the priority is defined as Conv+Bn greater than Conv+Activation, and the current operator is Conv while the other operator is Bn, then after step S203, the result should be yes, meaning that the current operator Conv and the other operator Bn conform to the preset fusion strategy, and the subsequent step S204 should be performed for fusion. However, if the priority is defined as Conv+Bn less than Conv+Activation, and the current operator is Conv while the other operator is Bn, and the current operator Conv is not formed by fusing the higher-priority Conv+Activation, then after step S203, the result should be no, meaning that the current operator Conv and the other operator Bn do not conform to the preset fusion strategy, and the subsequent step should proceed to step S205 instead of step S204.
[0038] S204: Merge the current operator and another operator into a new operator, and use the merged new operator as the current operator, and proceed to step S206.
[0039] Specifically, when the current operator and another operator are merged into a new operator, the current operator is replaced by the new operator, and the other operator is deleted from the initial image processing model structure; that is, when the model structure is visualized, the layer containing the other operator is no longer present. Furthermore, the operator type of the new operator formed by merging the current operator and another operator is the same as the current operator. For example, if the current operator is the convolution operator Conv, then the new operator is also the convolution operator Conv.
[0040] In one implementation, the specific process of merging the current operator and another operator into a new operator in step S204 above can include multiple cases, as follows:
[0041] Case A: In response to the current operator being the convolution operator Conv and the other operator being the normalization operator Bn, the convolution kernel and bias of the current operator are adjusted based on the normalization operator Bn to obtain a new fused operator.
[0042] In detail, the calculation of the convolution operator Conv can be abstracted as: y = w * x + b; where x represents the input, w represents the convolution kernel, b represents the bias, and y represents the output.
[0043] The computation of the normalization operator Bn can be abstracted as follows: Where y represents the input, u represents the mean of the input, and σ represents the mean of the input. 2 ε represents the variance of the input, ε represents a minimum value to prevent the denominator from being zero, α represents the scaling factor, and β represents the bias.
[0044] At this point, the new operator obtained by fusing the calculation of the normalization operator Bn into the convolution operator Conv can be abstracted as follows: Here Consider it as a new convolution kernel. This is the new bias. As you can see, the kernel weights and biases of the new Conv after fusion of Bn have changed compared to before fusion.
[0045] Case B: In response to the current operator being the convolution operator Conv and the other operator being the activation operator Activation, the computation conditions of the current operator are adjusted based on the activation operator Activation to obtain a new fused operator.
[0046] In detail, common activation operators include sigmoid and ReLU. Taking the ReLU layer as an example, the calculation principle of the ReLU layer is: when the input value is greater than or equal to 0, the output value is equal to the input value; and when the input value is less than 0, the input is 0. In existing technologies, the calculation processes of Conv operations and activation operations are independent. During inference, the Conv layer needs to be calculated first, and then the output of the Conv layer needs to be stored; then the previously saved output is read, the ReLU layer is calculated again, and the ReLU layer calculation result is stored back. As you can see, this is actually repeatedly accessing the same memory.
[0047] In this application, if Conv and ReLU are fused, that is, if the computational conditions of ReLU are directly introduced into the computation of Conv, a new fused operator is obtained; then the new fused operator can be abstracted as follows: Here, x represents the input, w represents the convolution kernel, b represents the bias, and y represents the output. During inference, one write-back operation is reduced; although the computational cost remains the same, memory access is decreased.
[0048] Case C: In response to the current operator being a convolution operator Conv and another operator being a computational function operator, adjust the convolution kernel and / or bias of the current operator based on the computational function operator to obtain a new fused operator.
[0049] In detail, the computational operators include Add, Sub, Mul, and Div operators.
[0050] The Add operator and Sub operator can be considered as one class. Taking the fusion of Conv and Add as an example, the add layer can be fused into the bias of Conv. The function of the add layer is to add corresponding elements of multiple inputs, and the bias of the Conv layer is the same operation. For example, if the output of the Conv layer is X, and X and another output Y are passed through the Add layer to get Z, the value of Y is accumulated into the bias of Conv to form a new operator after fusion. The new operator can be abstracted as: y = w*x + b + shift; where b represents the bias of Conv itself, shift represents another input of the original Add layer, and b + shift is used as the bias of the new Conv after fusion.
[0051] In a specific application scenario, assuming the data output format of each layer is NCHW, the Add layer has two inputs: one is the output of Conv, and the other is the output of other layers. It's important to note that since the bias length of Conv is the same as the number of channels in Conv (meaning each channel has the same bias value), when fusing Conv and Add, it's crucial to ensure that the length of the H and W dimensions of the other input to the Add layer is 1, or that the H and W values are obtained through broadcasting, meaning that the values in the H and W dimensions within the same C dimension are identical.
[0052] The multiplication operator Mul and the division operator Div can be considered as one class. Taking the fusion of Conv and Mul as an example, the Mul layer can be fused into the convolution kernel and bias of Conv. The role of the Mul layer is to multiply the corresponding elements of multiple inputs. The new operator formed after the fusion can be abstracted as: y = w*λ*x + b*λ; where λ represents another input of the original Mul layer, b*λ is used as the bias of the new Conv after fusion, and w*λ is used as the convolution kernel of the new Conv after fusion.
[0053] Case D: In response to the current operator being the Shape operator and the other operator being the extraction function operator, adjust the element values of the diagonal matrix corresponding to the current operator based on the data dimensions required to be extracted by the extraction function operator to obtain a new fused operator.
[0054] In detail, the Shape operator outputs the shape of the input tensor, and is usually followed by operators for extraction, concatenation, and computation. Assuming the input data of the Shape operator is in NCHW format, it can be represented in matrix form as: matrix A = [NCHW]. The Shape operator can be abstracted as a diagonal matrix M as shown below, and the result of multiplying matrix A by matrix M is the same as the original.
[0055] in,
[0056] In one application scenario, suppose we need to merge Shape and extraction function operators (Gather or Slice), and the extraction function operator only needs to extract data from the first to the third dimension. Then we can change the values of the diagonal elements on the diagonal matrix of the original Shape operator. The new operator after the change, that is, the merged one, can be abstracted as the diagonal matrix N shown below.
[0057] in,
[0058] Case E: In response to the current operator being the Shape operator and the other operator being the Concat concatenation operator, the diagonal matrix corresponding to the current operator is reduced, and elements related to the dimensions of the two sets of data to be concatenated by the concatenation operator are retained to obtain a new fused operator.
[0059] In detail, suppose Concat has two inputs: input matrix A has valid data only in dimension H, and input matrix B has valid data only in dimension W. The two inputs are processed by equivalent shape operator matrices M = [0 0 1 0] and N = [0 0 01], respectively, and the output is equal to the input. Concat concatenates the outputs of the two shape operators. An equivalent approach is to directly concatenate matrices M and N to obtain matrix L. Matrix L is used as the equivalent matrix of the shape operator, and the Concat layer is removed, thus achieving the fusion of the shape and concat layers.
[0060] in,
[0061] Case F: In response to the current operator being the Shape operator and the other operator being the computation function operator, a new row is added to the diagonal matrix corresponding to the current operator, and the element values on the diagonal of the diagonal matrix are adjusted or the element values on the new row are added based on the computation function operator to obtain the new merged operator.
[0062] In detail, the computational function operators include Add, Sub, Mul, and Div operators. To ensure compatibility with Add, Sub, Mul, and Div, a new row can be added to the equivalent matrix M of the Shape operator. The Shape operator can then be represented as matrix B, and the other input to the computational function operator can be represented as matrix A = [NCHW 1].
[0063] in,
[0064] Furthermore, the multiplication operator Mul and the division operator Div can be regarded as a class. Taking the fusion with the multiplication operator Mul as an example, the new operator after the fusion of the Shape operator and the multiplication operator Mul can be represented as the following matrix C;
[0065] in,
[0066] The Add operator and the Sub operator can be considered as one class; taking the fusion with Add as an example, assuming that we need to add 2 in the H dimension, the new operator after fusion of Shape operator and Add can be represented as the following matrix D;
[0067] in,
[0068] S205: Obtain the next operator from the initial image processing model, use the next operator as the current operator, and proceed to step S206.
[0069] Specifically, the next operator can be another operator connected to the output of the current operator in the previous step. Alternatively, when the current operator is the last operator on its branch, the next operator can be an operator on another branch.
[0070] S206: Return to step S202 (i.e., return to the step of determining whether the current operator is a preset operator and whether the number of branches connected to the output of the current operator is one).
[0071] Through the above steps S201-S206, all the operators that can be fused in the initial image processing model can be fused to obtain a more concise first fused image processing model.
[0072] S102: Obtain the first similarity between the first inference result and the second inference result; wherein, the first inference result is obtained by the initial image processing model inferring from the first image data, and the second inference result is obtained by the first fusion image processing model inferring from the first image data.
[0073] Specifically, the step of obtaining the first reasoning result can be located before or after the above step S101; and the above first similarity can be understood as cosine similarity.
[0074] S103: In response to a first similarity greater than or equal to a threshold, the first fusion image processing model is output so that the camera can use the first fusion image processing model to perform inference on the video frame to obtain the corresponding inference result.
[0075] Specifically, the threshold can be 1, and the specific threshold can be set according to the actual situation.
[0076] As can be seen, in the image processing method provided in this application, after the initial image processing model is fused by operators to obtain the first fused image processing model, the initial image processing model and the first fused image processing model are used to infer the same first data to obtain the first inference result and the second inference result respectively. Subsequently, the first fused image processing model is tested based on the first similarity between the first inference result and the second inference result, so that the final output first fused image processing model is simplified while the accuracy of the inference result is high.
[0077] In addition, in other embodiments, please refer to Figure 4 , Figure 4 for Figure 1A flowchart illustrating another implementation method following step S102, which includes the following steps performed concurrently with step S103:
[0078] S301: In response to a first similarity less than a threshold, obtain all fused operator combinations associated with the first fused image processing model.
[0079] Specifically, in the previous step S101, a corresponding operator fusion log is generated for every two operators fused; the above step S301 can obtain all fused operator combinations related to the first fused image processing model based on the stored log.
[0080] For example, all the merged operator combinations obtained include: Conv+Bn, Conv+Activation, and Shape+Add.
[0081] S302: For each combination of fused operators, make the preset fusion strategy contain only the current combination of fused operators, and re-execute the step of fusing at least two adjacent operators in the initial image processing model to obtain the fused second fused image processing model.
[0082] Specifically, it can be abstracted that each fusionable operator combination in each preset fusion strategy is equipped with a switch; the meaning of step S302 above is that only the switch corresponding to each fusionable operator combination is turned on in sequence; that is, each time the step of fusing at least two adjacent operators in the initial image processing model is re-executed, only one switch is turned on, and layer fusion will only be performed when the fusionable operator combination corresponding to the turned-on switch appears.
[0083] For example, when all the obtained fused operator combinations include Conv+Bn, Conv+Activation, and Shape+Add, the switches corresponding to Conv+Bn, Conv+Activation, and Shape+Add can be turned on individually to obtain the fused second fused image processing model.
[0084] Furthermore, the specific process of re-executing the step of fusing at least two adjacent operators in the initial image processing model is similar to... Figure 2 Similar to the case in China, I will not repeat it here.
[0085] S303: Obtain the second similarity between the first inference result and each third inference result; wherein the third inference result is obtained by inferring the first image data by the corresponding second fusion image processing model.
[0086] S304: In response to the second similarity being less than the threshold, it is determined that the initial image processing model has an anomaly when performing the combination of fused operators related to the second similarity.
[0087] That is, through the above steps S301-S304, the anomalous combinations of fusionable operators in the current initial image processing model can be located. All fused operator combinations related to the first fused image processing model are defined as a first combination set. Subsequently, after step S304, the fused operator combinations identified as anomalous can be removed from the first combination set, and the preset fusion strategy includes the removed first combination set. Then, the step of fusing at least two adjacent operators in the initial image processing model is re-executed to obtain a fused third fused image processing model; this third fused image processing model can then be used for inference acceleration.
[0088] In summary, this application provides an image processing method that first defines an operator fusion strategy and generates a fusion whitelist (i.e., fusion operator combinations). Each fusion operator combination corresponds to a switch, which can be turned on or off. An initial image processing model is loaded and forward inference is performed to obtain a first inference result. Then, a depth-first traversal is performed on the initial image processing model. If the model structure matches the network structure in the fusion whitelist, the old network structure is fused into a new simplified structure, and the fusion log is saved for backtracking. After the traversal is complete, the simplified first fusion image processing model is inferred to obtain a second inference result. The second inference result and the first inference result are compared using cosine similarity. If the similarity is 1, it indicates that the output results before and after image processing model optimization are equivalent, and the simplified first fusion image processing model is saved for subsequent inference tasks. If the similarity is not 1, it prompts the user to analyze the log to see if certain fusion operator combinations have bugs in the current image processing model structure. Combined with the fusion switch, the user is then asked to identify which fusion operator combinations caused the model's consistency problem for correction.
[0089] The image processing method provided in this application is described below from the perspective of the camera. For details, please refer to [link / reference needed]. Figure 5 , Figure 5 This is a flowchart illustrating another embodiment of the image processing method of this application, which includes:
[0090] S401: The first fusion image processing model obtained by the camera from the processor output.
[0091] The specific process of the processor outputting the first fused image processing model can be found in the above embodiments, and will not be repeated here.
[0092] S402: Use the first fusion image processing model to perform inference on the video frames to obtain the corresponding inference results.
[0093] Optionally, when the first fusion image processing model is an object detection model, the corresponding inference result is the object detection result; when the first fusion image processing model is an object tracking model, the corresponding inference result is the object tracking result; when the first fusion image processing model is an image classification model, the corresponding inference result is the image classification result.
[0094] In summary, by porting the smaller and more accurate first fusion image processing model output by the processor to the camera, the speed and accuracy of the camera in performing the corresponding tasks can be improved.
[0095] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an embodiment of the image processing apparatus of this application, which can be understood as a module diagram of the processor side. The image processing apparatus includes:
[0096] The first fusion module 10 is used to fuse at least two adjacent operators in the initial image processing model to obtain the fused first fused image processing model.
[0097] The first obtaining module 12, connected to the first fusion module 10, is used to obtain a first similarity between a first inference result and a second inference result; wherein, the first inference result is obtained by the initial image processing model inferring from the first image data, and the second inference result is obtained by the first fusion image processing model inferring from the first image data.
[0098] The first response module 14, connected to the first acquisition module 12, is used to output the first fused image processing model in response to the first similarity being greater than or equal to a threshold.
[0099] Please see Figure 7 , Figure 7This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device includes a memory 20 and a processor 22 coupled to each other. The memory 20 stores program instructions, and the processor 22 executes the program instructions to implement any of the above-described image processing methods. Specifically, the electronic device includes, but is not limited to, desktop computers, laptops, tablets, servers, etc., and is not limited thereto. Furthermore, the processor 22 can also be called a CPU (Center Processing Unit). The processor 22 may be an integrated circuit chip with signal processing capabilities. The processor 22 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. Additionally, the processor 22 can be implemented using integrated circuit chips.
[0100] Please see Figure 8 , Figure 8 This is a schematic diagram of a storage device according to an embodiment of the present application. The storage device 30 stores program instructions 300 that can be executed by a processor. The program instructions 300 are used to implement any of the above-described image processing methods. The storage device includes various media capable of storing program code, such as a USB flash drive, a portable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0101] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. An image processing method, characterized in that, include: The camera obtains a first fused image processing model output by the processor; wherein the process of the processor outputting the first fused image processing model includes: fusing at least two adjacent operators in the initial image processing model to obtain a fused first fused image processing model; obtaining a first similarity between a first inference result and a second inference result; wherein the first inference result is obtained by the initial image processing model inferring from the first image data, and the second inference result is obtained by the first fused image processing model inferring from the first image data; and outputting the first fused image processing model in response to the first similarity being greater than or equal to a threshold. The first fusion image processing model is used to infer the video frames to obtain the corresponding inference results. The step of fusing at least two adjacent operators in the initial image processing model to obtain a fused first image processing model includes: Starting from the input of the initial image processing model, one operator is obtained sequentially; Determine whether the current operator is a preset operator and whether the number of branches connected to the output of the current operator is one; If so, in response to the current operator and another operator connected to the output of the current operator conforming to a preset fusion strategy, the steps of merging the current operator and the other operator into a new operator and using the merged new operator as the current operator include: in response to the current operator being a shape operator and the other operator being an extraction function operator, adjusting the element values of the diagonal matrix corresponding to the current operator based on the data dimensions to be extracted by the extraction function operator to obtain the merged new operator; in response to the current operator being a shape operator and the other operator being a splicing function operator, reducing the diagonal matrix corresponding to the current operator and retaining elements related to the dimensions of the two sets of data to be spliced by the splicing function operator to obtain the merged new operator; in response to the current operator being a shape operator and the other operator being a calculation function operator, adding a row to the diagonal matrix corresponding to the current operator, and adjusting the element values on the diagonal of the diagonal matrix or adding a row based on the calculation function operator to obtain the merged new operator.
2. The image processing method according to claim 1, characterized in that, After determining whether the current operator is a preset operator and whether the number of branches connected to the output of the current operator is one, the method further includes: If so, in response to the current operator and another operator connected to the output of the current operator not conforming to the preset fusion strategy, the next operator is obtained from the initial image processing model, and the next operator is used as the current operator; wherein, the preset fusion strategy includes at least one combination of fusionable operators; If not, obtain the next operator from the initial image processing model and use the next operator as the current operator; Return to the step of determining whether the current operator is a preset operator and whether the number of branches connected to the output of the current operator is one.
3. The image processing method according to claim 1, characterized in that, The number of preset operators is multiple, and each preset operator corresponds to a preset fusion strategy that contains multiple fusionable operator combinations, and different fusionable operator combinations have different priorities; Prior to the step of responding to the current operator and another operator connected to the output of the current operator conforming to a preset fusion strategy, the following steps are included: Obtain the first priority of the fusionable operator combination corresponding to the current operator and the other operator; Determine whether the first priority is the highest priority, or whether the first priority is not the highest priority, and whether the current operator is formed by the fusion of fusionable operators with a priority greater than or equal to the first priority.
4. The image processing method according to claim 3, characterized in that, The preset operator is a convolution operator, and the preset fusion strategy corresponding to the convolution operator includes a fusionable operator combination consisting of convolution operators and normalization operators with decreasing priorities, a fusionable operator combination consisting of convolution operators and activation operators, and a fusionable operator combination consisting of convolution operators and computation function operators.
5. The image processing method according to claim 4, characterized in that, The step of fusing the current operator and the other operator into a new operator in response to a preset fusion strategy conforming to the current operator and the output of the current operator includes: In response to the current operator being a convolution operator and the other operator being a normalization operator, the convolution kernel and bias of the current operator are adjusted based on the normalization operator to obtain a new fused operator; In response to the current operator being a convolution operator and the other operator being an activation operator, the calculation conditions of the current operator are adjusted based on the activation operator to obtain a new fused operator; In response to the current operator being a convolution operator and the other operator being a computational function operator, the convolution kernel and / or bias of the current operator are adjusted based on the computational function operator to obtain a new fused operator.
6. The image processing method according to claim 3, characterized in that, The preset operator is a shape operator, and the preset fusion strategy corresponding to the shape operator includes a fusionable operator combination consisting of shape operators and extraction function operators with decreasing priority, a fusionable operator combination consisting of shape operators and splicing function operators, and a fusionable operator combination consisting of shape operators and calculation function operators.
7. The image processing method according to claim 2, characterized in that, Also includes: In response to the first similarity being less than a threshold, all fused operator combinations related to the first fused image processing model are obtained; For each of the fused operator combinations, the preset fusion strategy is made to include only the current fused operator combination, and the step of fusing at least two adjacent operators in the initial image processing model is re-executed to obtain the fused second fused image processing model. A second similarity is obtained between the first inference result and each third inference result; wherein the third inference result is obtained by inferring from the first image data by the corresponding second fusion image processing model; In response to the second similarity being less than a threshold, it is determined that the initial image processing model has an anomaly when performing the combination of the fused operators related to the second similarity.
8. An image processing method, characterized in that, include: The step of the processor fusing at least two adjacent operators in the initial image processing model to obtain a fused first image processing model includes: sequentially obtaining an operator starting from the input end of the initial image processing model; determining whether the current operator is a preset operator and whether the number of branches connected to the output end of the current operator is one; if so, in response to the current operator and another operator connected to the output end of the current operator conforming to a preset fusion strategy, fusing the current operator and the other operator into a new operator, and using the fused new operator as the current operator, the step includes: in response to the current operator being a shape operator and the other operator being an extraction function operator, based on the extraction... The following steps are taken: First, adjust the element values of the diagonal matrix corresponding to the current operator to obtain a new fused operator. Second, if the current operator is a shape operator and the other operator is a concatenation operator, reduce the diagonal matrix corresponding to the current operator and retain elements related to the dimensions of the two sets of data to be concatenated by the concatenation operator to obtain a new fused operator. Third, if the current operator is a shape operator and the other operator is a computation operator, add a row to the diagonal matrix corresponding to the current operator and adjust the element values on the diagonal of the diagonal matrix or add a row based on the computation operator to obtain a new fused operator. A first similarity is obtained between a first inference result and a second inference result; wherein the first inference result is obtained by the initial image processing model inferring from the first image data, and the second inference result is obtained by the first fusion image processing model inferring from the first image data; In response to the first similarity being greater than or equal to a threshold, the first fusion image processing model is output so that the camera can use the first fusion image processing model to perform inference on video frames to obtain the corresponding inference results.
9. A storage device, characterized in that, The system stores program instructions that can be executed by a processor, the program instructions being used to implement the image processing method according to any one of claims 1 to 8.