Training method and processing method, device and equipment based on image processing model

Abstract

This application provides a training method, processing method, apparatus, and device based on an image processing model, comprising: acquiring a training image; inputting the training image into a preset initial model; performing convolution processing on the training image based on a first convolution kernel of a convolutional layer in the initial model to obtain a first output vector; and performing convolution processing on the training image based on a second convolution kernel of a convolutional layer in the initial model to obtain a second output vector; and determining a third convolution kernel based on the first and second output vectors to obtain an image processing model with a third convolution kernel. This application achieves improved expressive power of the convolutional layer with a third convolution kernel while maintaining the same computational speed and resource consumption, enabling the convolutional layer to analyze target data from multiple perspectives and obtain more accurate analysis results.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a training method, processing method, apparatus and device based on an image processing model. Background Technology

[0002] Image processing models based on convolutional neural networks are a core technology in the field of computer vision. As a general-purpose architecture, they can be directly applied to downstream tasks such as classification, detection, segmentation, and retrieval, as well as scenarios such as autonomous driving, smart cities, and industrial fault detection.

[0003] Currently, machine learning methods are commonly used to directly train the convolutional kernels of image processing models to obtain mature image processing models. However, the inventors have discovered that the convolutional kernels obtained solely through machine learning methods have relatively limited training dimensions, resulting in low expressive power and consequently reducing the computational accuracy of mature neural networks. Summary of the Invention

[0004] This application provides a training method, processing method, apparatus, and device based on an image processing model to solve the problem that the low expressive power of convolution kernels caused by current machine learning methods results in low computational accuracy of the mature image processing model.

[0005] Firstly, this application provides a training method based on an image processing model, comprising:

[0006] Obtain the image to be trained;

[0007] The image to be trained is input into a preset initial model. The image is then convolved using the first convolutional kernel of the convolutional layer in the initial model to obtain a first output vector. The image is then convolved using the second convolutional kernel of the convolutional layer in the initial model to obtain a second output vector. The first output vector represents the image processing result generated by machine learning on the image based on the first convolutional kernel. The second output vector represents the image processing result generated by machine learning on the image based on the second convolutional kernel.

[0008] Based on the first output vector and the second output vector, a third convolution kernel is determined for the first convolution kernel and the second convolution kernel to obtain an image processing model with the third convolution kernel; wherein, the image processing model is used to process the image to be processed to obtain the image processing result of the image to be processed.

[0009] In the above scheme, the step of performing convolution processing on the image to be trained based on the second convolution kernel of the convolutional layer in the initial model to obtain the second output vector includes:

[0010] The training image is convolved based on the occlusion kernel in the second convolution kernel to obtain occlusion output data.

[0011] The occlusion output data is convolved based on the unit kernel corresponding to the occlusion kernel in the second convolution kernel to obtain the second output vector.

[0012] In the above scheme, determining the third convolution kernel based on the first output vector and the second output vector, including:

[0013] If it is determined that the difference between the first output vector and the training label of the image to be trained is less than a preset first training threshold, and the difference between the second output vector and the training label of the image to be trained is less than a preset second training threshold, then the first convolutional kernel and the second convolutional kernel are reparameterized to obtain the third convolutional kernel; wherein, the reparameterization is used to integrate the weights in at least one convolutional kernel to form a convolutional kernel; the first training threshold reflects that the first convolutional kernel that generates the first output vector based on the image to be trained is capable of image processing on the image to be trained and obtaining an image processing result matching the training label; the second training threshold reflects that the second convolutional kernel that generates the second output vector based on the image to be trained is capable of image processing on the image to be trained and obtaining an image processing result matching the training label.

[0014] In the above scheme, the step of reparameterizing the first convolutional kernel and the second convolutional kernel to obtain the third convolutional kernel includes:

[0015] The occlusion kernel and the corresponding unit kernel in the second convolution kernel are reparameterized to obtain a second reparameter kernel; wherein, the occlusion kernel is a convolution kernel obtained by occluding and copying one or more weights in the first convolution kernel; the occlusion kernel is used to perform convolution processing on the image to be trained to obtain an occlusion output vector; the unit kernel is used to perform convolution processing on the occlusion output vector to obtain a second output vector.

[0016] The first convolution kernel and the second multi-parameter kernel are reparameterized to obtain the third convolution kernel.

[0017] In the above scheme, before inputting the image to be trained into the preset initial model, the method further includes:

[0018] A second convolutional kernel corresponding to the first convolutional kernel is created in the convolutional layer of the initial model; wherein the second convolutional kernel is arranged in parallel with the first convolutional kernel; the second convolutional kernel includes an occlusion kernel and a unit kernel corresponding to the occlusion kernel, the occlusion kernel and the unit kernel are arranged in series; the occlusion kernel is used to perform convolution processing on the image to be trained to obtain an occlusion output vector; the unit kernel is used to perform convolution processing on the occlusion output vector to obtain a second output vector.

[0019] In the above scheme, creating a second convolutional kernel corresponding to the first convolutional kernel in the convolutional layer of the initial model includes:

[0020] According to the preset occlusion rules, one or more weights in the copied first convolution kernel are converted into preset occlusion values ​​to obtain an occlusion kernel corresponding to the first convolution kernel.

[0021] By concatenating the occlusion kernel and the preset unit kernel, a second convolution kernel corresponding to the first convolution kernel is obtained.

[0022] The above scheme further includes, before obtaining the image processing model with the third convolutional kernel:

[0023] The image to be inferred is obtained, and the image to be inferred is convolved based on the third convolution kernel of the convolutional layer in the initial model to obtain a third output vector; wherein, the third output vector is used to characterize the image processing result generated by image processing of the image to be inferred based on the third convolution kernel;

[0024] If it is determined that the difference between the third output vector and the inference label of the image to be inferred is greater than a preset inference threshold, then the initial model is trained using the image to be inferred to obtain the image processing model.

[0025] Secondly, this application provides an image processing method based on an image processing model, including:

[0026] The image to be processed is obtained, and the image to be processed is input into a preset image processing model. The image to be processed is convolved based on the third convolution kernel of the convolutional layer in the image processing model to obtain the image processing result corresponding to the image to be processed.

[0027] The image processing model is the same as the image processing model described in the first aspect.

[0028] Thirdly, this application provides a training device based on an image processing model, comprising:

[0029] The training input module is used to acquire the image to be trained;

[0030] The training module is used to input the image to be trained into a preset initial model, perform convolution processing on the image to be trained based on the first convolution kernel of the convolutional layer in the initial model to obtain a first output vector, and perform convolution processing on the image to be trained based on the second convolution kernel of the convolutional layer in the initial model to obtain a second output vector; wherein, the first output vector is used to represent the image processing result generated by machine learning on the image to be trained based on the first convolution kernel; the second output vector is used to represent the image processing result generated by machine learning on the image to be trained based on the second convolution kernel;

[0031] The model generation module is used to determine a third convolution kernel based on the first output vector and the second convolution kernel, so as to obtain an image processing model with the third convolution kernel; wherein the image processing model is used to process the image to be processed to obtain the image processing result of the image to be processed.

[0032] Fourthly, this application provides an image processing apparatus based on an image processing model, comprising:

[0033] The input processing module is used to acquire the image to be processed and input the image to be processed into a preset image processing model;

[0034] The image processing module is used to perform convolution processing on the image to be processed based on the third convolution kernel of the convolution layer in the image processing model, so as to obtain the image processing result corresponding to the image to be processed; wherein, the image processing model is the image processing model in the above-mentioned device.

[0035] Fifthly, this application provides a computer device, including: a processor and a memory communicatively connected to the processor;

[0036] The memory stores computer-executed instructions;

[0037] The processor executes computer execution instructions stored in the memory to implement the above method.

[0038] Sixthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the above-described method.

[0039] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0040] This application provides a training method, processing method, apparatus, and device based on an image processing model. The method involves inputting the image to be trained into a preset initial model, performing convolution processing on the image based on the first convolution kernel of the convolutional layer in the initial model to obtain a first output vector, and performing convolution processing on the image based on the second convolution kernel of the convolutional layer in the initial model to obtain a second output vector. Machine learning is then performed on the first and second convolution kernels respectively to achieve multi-dimensional training of the convolutional layer, thereby improving the overall expressive power of the convolutional layer.

[0041] By obtaining a third convolutional kernel based on the first and second convolutional kernels, this third convolutional kernel, being based on multi-dimensional training, possesses higher expressive power. Furthermore, since the kernel size of the third convolutional kernel is the same as that of the first convolutional kernel, the computational speed and computational resources consumed by the third and first convolutional kernels are identical. This achieves improved expressive power of the convolutional layer while maintaining the same computational speed and computational resources consumed, enabling the convolutional layer to analyze target data from multiple perspectives and obtain more accurate analysis results. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] Figure 1 A schematic diagram of the structure of the initial model when creating the second convolution kernel and the transfer kernel in an application scenario provided in this application embodiment;

[0044] Figure 2 This application provides a schematic diagram of the image processing model structure when obtaining the third convolution kernel in an application scenario.

[0045] Figure 3 A flowchart of Embodiment 1 of a training method based on an image processing model provided in this application;

[0046] Figure 4 In Embodiment 1 of an image processing model training method provided in this application, a schematic diagram is shown of the process of converting the first convolution kernel into an occlusion kernel through occlusion rules.

[0047] Figure 5 A flowchart of Embodiment 2 of an image processing method based on an image processing model provided in this application;

[0048] Figure 6 A schematic diagram of the program modules of an image processing device based on an image processing model provided by the present invention, in embodiment 3;

[0049] Figure 7 A schematic diagram of the program modules of an embodiment 4 of an image processing device based on an image processing model provided by the present invention;

[0050] Figure 8 This is a schematic diagram of the hardware structure of the computer device in the computer device of the present invention.

[0051] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0053] The specific application scenario for this application is as follows:

[0054] Please see Figure 1 A training method based on an image processing model is run in the initial model 3. The initial model 3 acquires the image to be trained; the image to be trained is input into the preset initial model 3, and convolution processing is performed on the image to be trained based on the first convolution kernel 32 of the convolutional layer 31 in the initial model 3 to obtain a first output vector, and convolution processing is performed on the image to be trained based on the second convolution kernel 33 of the convolutional layer 31 in the initial model 3 to obtain a second output vector; wherein, the second convolution kernel 33 includes at least one occlusion kernel 331 and at least one unit kernel 332, the second convolution kernel 33 is arranged in parallel with the first convolution kernel 32, and each occlusion kernel 331 in the second convolution kernel 33 is arranged in series with each unit kernel 332. Further, a transfer kernel 34 corresponding to the first convolution kernel 32 is created, and the transfer kernel 34 is arranged in parallel with both the first convolution kernel 32 and the second convolution kernel 33.

[0055] Please see Figure 2Based on the first and second output vectors, a third convolution kernel 35 is determined from the first convolution kernel 32 and the second convolution kernel 33 to obtain an image processing model with a third convolution kernel. This image processing model is used to process the image to be processed, obtaining the image processing result. The transfer kernel 34 in the convolutional layer 3 with the third convolution kernel 35 is deleted. The image processing model 3 with the third convolution kernel 35 improves the expressive power of the convolutional layer while maintaining its computational speed and resource consumption. This allows the convolutional layer to analyze target data from multiple perspectives and obtain more accurate analysis results, thereby improving the computational accuracy of the image processing model with the third convolution kernel 35.

[0056] Image processing models are based on neural networks. Neural networks explore and simulate the functions of the human brain's nervous system by modeling and connecting neurons—the basic units of the human brain—and developing an artificial system with intelligent information processing functions such as learning, association, memory, and pattern recognition. The neural network in this application is a convolutional neural network (CNN) for computer vision. A CNN is a feedforward neural network whose artificial neurons can respond to surrounding units within a certain coverage area, exhibiting excellent performance in large-scale image processing. A CNN consists of one or more convolutional layers and a fully connected layer at the top (corresponding to a classic neural network), and also includes associated weights and pooling layers.

[0057] The technical solution of this application and how the technical solution of this application solves the problems of the prior art will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0058] Example 1:

[0059] Please see Figure 3 This application provides a training method based on an image processing model, including:

[0060] S101: Obtain the image to be trained.

[0061] In this step, the training image has a preset training label, which represents the image type of the training image, or the object type of the object in the training image, or the target object in the training image (e.g., the processing method for extracting the target object from the training image).

[0062] S102: Input the image to be trained into a preset initial model, perform convolution processing on the image to be trained based on the first convolution kernel of the convolutional layer in the initial model to obtain a first output vector, and perform convolution processing on the image to be trained based on the second convolution kernel of the convolutional layer in the initial model to obtain a second output vector; wherein, the first output vector is used to represent the image processing result generated by machine learning on the image to be trained based on the first convolution kernel; the second output vector is used to represent the image processing result generated by machine learning on the image to be trained based on the second convolution kernel;

[0063] In this step, machine learning is performed on the first and second convolutional kernels to train the convolutional layer from multiple dimensions, thereby improving the overall expressive power of the convolutional layer.

[0064] Machine learning is a process that uses the simulation or realization of human learning behavior to train an image processing model to reorganize the existing knowledge structure in the image processing model and obtain new knowledge or skills that can obtain image processing results based on the image to be trained.

[0065] In a preferred embodiment, before inputting the image to be trained into a preset initial model, the method further includes:

[0066] In the convolutional layer of the initial model, a second convolutional kernel corresponding to the first convolutional kernel is created; wherein, the second convolutional kernel is arranged in parallel with the first convolutional kernel; the second convolutional kernel includes an occlusion kernel and a unit kernel corresponding to the occlusion kernel, and the occlusion kernel and the unit kernel are arranged in series; the occlusion kernel is used to perform convolution processing on the image to be trained to obtain an occlusion output vector; the unit kernel is used to perform convolution processing on the occlusion output vector to obtain a second output vector.

[0067] Specifically, a second convolutional kernel corresponding to the first convolutional kernel is created in the convolutional layer of the initial model, including:

[0068] According to the preset occlusion rules, one or more weights in the copied first convolution kernel are converted into preset occlusion values ​​to obtain the occlusion kernel corresponding to the first convolution kernel;

[0069] By concatenating the occlusion kernel and the preset unit kernel, a second convolution kernel corresponding to the first convolution kernel is obtained.

[0070] In this example, the occlusion rule is a template used to convert at least one weight in the copied first convolutional kernel into an occlusion value. For example: See [link to relevant documentation]. Figure 4The occlusion value can be set to 0. If the first convolutional kernel is a 3×3 kernel, an occlusion rule is to convert the weights of coordinates (1,1), (2,2), and (3,3) to an occlusion value of 0, resulting in a 3×3 convolutional kernel with weights of 0 at coordinates (1,1), (2,2), and (3,3). Therefore, by generating occlusion kernels based on the first convolutional kernel using occlusion rules, the technical effect of automatically generating multiple occlusion kernels is achieved. In this embodiment, the unit kernel is a 1×1 convolutional kernel.

[0071] In a preferred embodiment, the training image is convolved based on the second convolutional kernel of the convolutional layer in the initial model to obtain a second output vector, including:

[0072] The training image is convolved based on the occlusion kernel in the second convolution kernel to obtain occlusion output data.

[0073] The occlusion output data is convolved based on the unit kernel corresponding to the occlusion kernel in the second convolution kernel to obtain the second output vector.

[0074] In this example, an occlusion kernel is set to train the convolutional layer from multiple dimensions. The occlusion kernel partially displays the weights of the first convolutional kernel by hiding one or more weights in the copied kernel, thus enabling machine learning of the obtained occlusion kernel from the dimension of hiding some weights and improving the expressive power of the convolutional layer.

[0075] During training, the homogeneity of different channels in the occlusion kernel continuously increases (i.e., the weights on different channels of the occlusion kernel become increasingly consistent). This example addresses this by setting unit kernels with different weights on different channels and mapping the weights of different channels on the unit kernel to the weights of different channels in the occlusion kernel. This yields the machine-learned occlusion kernel, and units whose machine-learned channel weights are mapped to the weights of each channel in the occlusion kernel. This eliminates the situation where machine learning is only performed on the occlusion kernel, leading to homogeneity in the weights of each channel of the obtained occlusion kernel. In this embodiment, the unit kernel is a 1×1 convolution kernel.

[0076] S103: Based on the first output vector and the second output vector, determine the third convolution kernel for the first convolution kernel and the second convolution kernel to obtain an image processing model with the third convolution kernel; wherein, the image processing model is used to process the image to be processed to obtain the image processing result of the image to be processed.

[0077] In this step, a third convolutional kernel is obtained based on the first and second convolutional kernels. This third convolutional kernel, being based on multi-dimensional training, has higher expressive power. Since the kernel size of the third convolutional kernel is the same as that of the first convolutional kernel, the computational speed and computational resources consumed by the third and first convolutional kernels are the same. Thus, while ensuring that the computational speed and computational resources consumed by the convolutional layer with the third convolutional kernel remain unchanged, the expressive power of the convolutional layer is improved, enabling the convolutional layer to analyze the target data from multiple perspectives and obtain more accurate analysis results.

[0078] In a preferred embodiment, determining a third convolution kernel based on the first and second output vectors includes:

[0079] If the difference between the first output vector and the training label of the image to be trained is less than a preset first training threshold, and the difference between the second output vector and the training label of the image to be trained is less than a preset second training threshold, then the first and second convolutional kernels are reparameterized to obtain a third convolutional kernel. The reparameterization process is used to integrate the weights in at least one convolutional kernel to form a single convolutional kernel. The first training threshold reflects that the first convolutional kernel, which generates the first output vector from the image to be trained, can perform image processing on the image to be trained and obtain an image processing result matching the training label. The second training threshold reflects that the second convolutional kernel, which generates the second output vector from the image to be trained, can perform image processing on the image to be trained and obtain an image processing result matching the training label.

[0080] In this example, re-parameterization is a method used to integrate two or more convolutional kernels with the same kernel size to obtain a single convolutional kernel. In this embodiment, re-parameterization is used to match and add the weights of two or more convolutional kernels one by one to obtain a single convolutional kernel.

[0081] Since the second convolutional kernel is obtained based on the modification of the first convolutional kernel, by setting a first training threshold and a second training threshold to define the requirements for the first and second convolutional kernels as training maturity states respectively, the training management of the two different convolutional kernels at different levels is realized, thus expanding the diversity of training of image processing models.

[0082] In this embodiment, based on the position of at least one weight in the first convolution kernel and the position of at least one weight in the second convolution kernel, at least one weight pair is obtained by corresponding the weights in the first convolution kernel and the weights in the second convolution kernel one-to-one; a summation operation, a product operation, a mean operation, or a weighted summation operation is performed on the weights in the at least one weight pair to obtain the operation weights corresponding to each weight pair; the operation weights are arranged according to the positions corresponding to the weight pairs to obtain the third convolution kernel.

[0083] Specifically, the first and second convolutional kernels are reparameterized to obtain the third convolutional kernel, which includes:

[0084] The second convolutional kernel is obtained by reparameterizing the occlusion kernel and the corresponding unit kernel. The occlusion kernel is obtained by copying one or more weights from the first convolutional kernel through occlusion. The occlusion kernel is used to perform convolution processing on the image to be trained to obtain the occlusion output vector. The unit kernel is used to perform convolution processing on the occlusion output vector to obtain the second output vector.

[0085] The first and second convolution kernels are reparameterized to obtain the third convolution kernel.

[0086] In this example, based on the positions of at least one weight in the occlusion kernel and at least one weight in the unit kernel, at least one weight pair is obtained by mapping the weights in the occlusion kernel to the weights in the unit kernel; a summation operation, a product operation, a mean operation, or a weighted summation operation is performed on the weights in the at least one weight pair to obtain the operation weight corresponding to each weight pair; the operation weights are arranged according to the positions corresponding to the weight pairs to obtain the second parameter kernel.

[0087] By training the occlusion kernel, machine learning is performed on the obtained occlusion kernel from the dimension of partially hidden weights, improving the expressive power of the convolutional layer. By training the unit kernel, different weights are set in different channels of the unit kernel. The second parameter kernel obtained by reparameterizing the unit kernel and the occlusion kernel can reflect the weights in the occlusion kernel, as well as the weights in the unit kernel that reflect the weights in the occlusion kernel. Therefore, the obtained inference reparameter kernel can effectively eliminate the situation where the weights of each channel of the obtained occlusion kernel tend to be homogeneous due to the current method of only performing machine learning on the occlusion kernel.

[0088] Based on the position of at least one weight in the first convolution kernel and the position of at least one weight in the second parameter kernel, at least one weight pair is obtained by matching the weights in the first convolution kernel with the weights in the second parameter kernel; a summation operation, a product operation, a mean operation, or a weighted summation operation is performed on the weights in the at least one weight pair to obtain the operation weights corresponding to each weight pair; the operation weights are arranged according to the positions corresponding to the weight pairs to obtain the third convolution kernel.

[0089] Optionally, after creating a second convolutional kernel corresponding to the first convolutional kernel in the convolutional layers of the initial model, the method further includes:

[0090] Create a transfer kernel corresponding to the first convolution kernel, and arrange the transfer kernel in parallel with the first convolution kernel and the second convolution kernel.

[0091] In this step, a 1×1 convolutional kernel is used as the learning transfer kernel. When the reverse iterator performs machine learning on the convolutional layers of the image processing model, it can bypass the first and second convolutional kernels of the current convolutional layer through the transfer kernel and train the convolutional layer located one position before the current convolutional layer. This allows for iterative adjustment of the weights in the first and second convolutional kernels of each convolutional layer in the image processing model, achieving comprehensive backpropagation of the reverse iterator and ensuring that each convolutional layer in the image processing model can be iteratively adjusted.

[0092] In a preferred embodiment, before obtaining the image processing model with the third convolutional kernel, the method further includes:

[0093] The image to be reasoned is obtained, and the image to be reasoned is convolved based on the third convolution kernel of the convolutional layer in the initial model to obtain the third output vector; wherein, the third output vector is used to represent the image processing result generated by image processing of the image to be reasoned based on the third convolution kernel;

[0094] If the difference between the third output vector and the inference label of the image to be inferred is determined to be greater than a preset inference threshold, the initial model is trained using the image to be inferred to obtain the image processing model.

[0095] In this example, the inference process involves inputting unknown data into a training model created through a learning process, and then classifying, identifying, and predicting the data based on the trained model.

[0096] By performing inference on an initial model with a third convolutional kernel before obtaining the image processing model, it is ensured that the final image processing model can obtain a third output vector that matches (is the same or similar to) the inference label of the image to be inferred, based on the image to be inferred other than the training image, thus ensuring the applicability of the image processing model.

[0097] The inference label represents the image type of the image to be inferred, the object type of the object in the training image, or the target object in the training image (e.g., the processing method for extracting the target object from the image to be trained).

[0098] The inference threshold reflects the ability of the third convolution kernel, which generates the third output vector from the image to be inferred, to perform image processing on the image to be inferred and obtain an image processing result that matches the inference label.

[0099] Optionally, after obtaining the image processing model with a third convolutional kernel, the following steps are also included:

[0100] Remove the transit kernel from the convolutional layer that has a third convolutional kernel.

[0101] This step removes the transit kernel to prevent the image processing model from bypassing the third convolutional kernel in the convolutional layer during use, thus preventing the convolutional layer from extracting features.

[0102] Example 2:

[0103] Please see Figure 5 This application also provides an image processing method based on an image processing model, comprising:

[0104] S201: Obtain the image to be processed and input the image to be processed into the preset image processing model;

[0105] S202: The third convolution kernel of the convolutional layer in the image processing model is used to perform convolution processing on the image to be processed, so as to obtain the image processing result corresponding to the image to be processed; wherein, the image processing model is the image processing model in Example 1.

[0106] In this example, an image processing model is used to process the image to be processed and obtain the image processing result. Compared with other existing image processing models, it has better recognition accuracy and faster running speed. Specifically, by adjusting the parameters of the convolution kernels in the image processing model, a first network model and a second network model can be obtained.

[0107] In a preferred embodiment, the image to be processed is convolved based on the third convolution kernel of the convolutional layer in the image processing model to obtain the image processing result corresponding to the image to be processed, including:

[0108] The third output vector is obtained by performing convolution processing on the image to be inferred based on the third convolution kernel of the convolutional layer in the image processing model.

[0109] The image processing result is generated by the fully connected layer in the image processing model performing image processing on the image to be processed based on the third output vector.

[0110] In this step, fully connected layers (FC) act as a "classifier" in the entire convolutional neural network. If operations such as convolutional layers, pooling layers, and activation functions map the raw data to the hidden feature space, then fully connected layers map the learned "distributed feature representation" to the sample label space.

[0111] For example, the image processing model includes a first network model and a second network model. The first network model is DLNet, and the second network model is DLGNet. The model parameters of the first network model and the second network model are shown in Table 1.

[0112] Table 1:

[0113]

[0114] Wherein, Conv1, Con2, Stage3, Stage4, Stage5, and Con6 are convolutional layers of the image processing model proposed in this application, each having at least one first convolutional kernel. Output Size is the output size of the convolutional layer, Kernel Size is the kernel size of the first convolutional kernel in the convolutional layer; Stride is the stride of the first convolutional kernel in the convolutional layer, Repeat is the number of times the first convolutional kernel in the convolutional layer repeats convolution; Groups is used to group the corresponding input channels and output channels, with a default value of 1, meaning that all input and output channels are grouped separately by default. For example: if the input data size is 90x100x100x32, where 90 is the batch size, 100x100 is the image data shape, and 32 is the number of channels, and it needs to go through a 3x3x48 convolution, the default group is 1, which is a fully connected convolutional layer. If group is 2, then the 32 input channels are divided into two 16-channel groups, and the 48 output channels are divided into two 24-channel groups. For the two 24-channel outputs, the first 24-channel is fully convolved with the first 16-channel input, and the second 24-channel is fully convolved with the second 16-channel input. In the extreme case where the number of input and output channels is the same, for example, 24, and the group size is also 24, then each output convolution kernel is only convolved with the corresponding channel of the input. `output_channels` is the number of output channels.

[0115] Furthermore, as shown in Table 2, the image processing model with a third convolutional kernel in this application is compared with the current reparameterization methods ACNet, RepVGG and DBB on the CIFAR-10 classification task and ImageNet classification task, with the Acc metric being the Acc metric of the image processing model with a third convolutional kernel in this application.

[0116] Table 2:

[0117]

[0118]

[0119] Wherein, ACNet is ACNet:

[0120] Strengthening the Kernel Skeletons for Powerful CNN via AsymmetricConvolution Blocks, ICCV 2019.

[0121] RepVGG is Making VGG-style ConvNets Great Again, CVPR 2021.

[0122] DBB is Diverse Branch Block: Building a Convolution as an Inception-Like Unit (CVPR 2021).

[0123] The Acc metric is a measure of machine learning accuracy.

[0124] Furthermore, as shown in Table 3, the second network model obtained in this application is compared with other image processing models in terms of Acc, speed, flops, and params. It can be found that the first and second network models obtained in this application, while achieving similar accuracy on classification tasks, have significantly higher speeds on the NPU than other methods.

[0125] Table 3:

[0126]

[0127] Furthermore, as shown in Table 4, the parameters of the second network model obtained in this application are compared with those of other mobile image processing models in PASCOL VOC object detection.

[0128] Table 4:

[0129]

[0130] Furthermore, as shown in Table 5, the parameters of the second network model obtained in this application are compared with those of other mobile image processing models in MS COCO object detection.

[0131] Table 5:

[0132]

[0133] Furthermore, as shown in Table 6, the parameters of the second network model obtained in this application are compared with those of other mobile image processing models in Cityscapes semantic segmentation.

[0134] Table 6:

[0135]

[0136] Therefore, as shown in Tables 4-6, the second network model obtained in this application has significant advantages over various mainstream lightweight image processing models in object detection and semantic segmentation tasks.

[0137] Example 3:

[0138] Please see Figure 6 This application also provides a training device 1 based on an image processing model, comprising:

[0139] Training input module 11 is used to acquire the image to be trained;

[0140] The training module 12 is used to input the image to be trained into a preset initial model, perform convolution processing on the image to be trained based on the first convolution kernel of the convolutional layer in the initial model to obtain a first output vector, and perform convolution processing on the image to be trained based on the second convolution kernel of the convolutional layer in the initial model to obtain a second output vector; wherein, the first output vector is used to represent the image processing result generated by machine learning on the image to be trained based on the first convolution kernel; the second output vector is used to represent the image processing result generated by machine learning on the image to be trained based on the second convolution kernel;

[0141] The model generation module 13 is used to determine a third convolution kernel based on the first output vector and the second convolution kernel, so as to obtain an image processing model with a third convolution kernel; wherein, the image processing model is used to process the image to be processed to obtain the image processing result of the image to be processed.

[0142] Preferably, the training module 12 includes:

[0143] The convolutional kernel creation unit 121 is used to create a second convolutional kernel corresponding to the first convolutional kernel in the convolutional layer of the initial model; wherein the second convolutional kernel is arranged in parallel with the first convolutional kernel; the second convolutional kernel includes an occlusion kernel and a unit kernel corresponding to the occlusion kernel, and the occlusion kernel and the unit kernel are arranged in series; the occlusion kernel is used to perform convolution processing on the image to be trained to obtain an occlusion output vector; the unit kernel is used to perform convolution processing on the occlusion output vector to obtain a second output vector.

[0144] The occlusion convolution unit 122 is used to perform convolution processing on the training image based on the occlusion kernel in the second convolution kernel to obtain occlusion output data.

[0145] The unit convolution unit 123 is used to perform convolution processing on the occlusion output data based on the unit kernel corresponding to the occlusion kernel in the second convolution kernel to obtain the second output vector.

[0146] Preferably, the convolution kernel creation unit 121 includes:

[0147] Kernel creation subunit 1211 is used to convert one or more weights in the copied first convolution kernel into preset occlusion values ​​according to preset occlusion rules, so as to obtain an occlusion kernel corresponding to the first convolution kernel;

[0148] The kernel arrangement subunit 1212 is used to connect the occlusion kernel and the preset unit kernel in series to obtain the second convolution kernel corresponding to the first convolution kernel.

[0149] Preferably, the model generation module 13 includes:

[0150] The reparameterization module 131 is used to perform reparameterization processing on the first convolutional kernel and the second convolutional kernel to obtain a third convolutional kernel if it is determined that the difference between the first output vector and the training label of the image to be trained is less than a preset first training threshold, and the difference between the second output vector and the training label of the image to be trained is less than a preset second training threshold. The reparameterization processing is used to integrate the weights in at least one convolutional kernel to form a single convolutional kernel. The first training threshold reflects that the first convolutional kernel, which generates the first output vector from the image to be trained, can perform image processing on the image to be trained and obtain an image processing result matching the training label. The second training threshold reflects that the second convolutional kernel, which generates the second output vector from the image to be trained, can perform image processing on the image to be trained and obtain an image processing result matching the training label.

[0151] The model inference module 132 is used to acquire the image to be inferred, and to perform convolution processing on the image to be inferred based on the third convolution kernel of the convolutional layer in the initial model to obtain the third output vector; wherein, the third output vector is used to represent the image processing result generated by image processing of the image to be inferred based on the third convolution kernel.

[0152] The inference training module 133 is used to train the initial model using the image to be inferred if the difference between the third output vector and the inference label of the image to be inferred is greater than a preset inference threshold, so as to obtain an image processing model.

[0153] Preferably, the reparameterization module 131 includes:

[0154] The occlusion unit reparameterization unit 1311 is used to reparameterize the occlusion kernel and the corresponding unit kernel in the second convolution kernel to obtain the second reparameter kernel; wherein, the occlusion kernel is a convolution kernel obtained by occluding and copying one or more weights in the first convolution kernel; the occlusion kernel is used to perform convolution processing on the image to be trained to obtain the occlusion output vector; the unit kernel is used to perform convolution processing on the occlusion output vector to obtain the second output vector;

[0155] The kernel reparameterization unit 1312 is used to reparameterize the first convolution kernel and the second reparameterization kernel to obtain the third convolution kernel.

[0156] Example 4:

[0157] Please see Figure 7 This application also provides an image processing apparatus 2 based on an image processing model, comprising:

[0158] The input processing module 21 is used to acquire the image to be processed and input the image to be processed into a preset image processing model.

[0159] Image processing module 22 is used to perform convolution processing on the image to be processed based on the third convolution kernel of the convolution layer in the image processing model, so as to obtain the image processing result corresponding to the image to be processed; wherein, the image processing model is the image processing model in embodiment 3.

[0160] Preferably, the image processing module 22 includes:

[0161] The convolution processing module 221 is used to perform convolution processing on the image to be reasoned based on the third convolution kernel of the convolutional layer in the image processing model to obtain the third output vector.

[0162] The fully connected processing module 222 is used to generate image processing results based on the third output vector of the fully connected layer in the image processing model to perform image processing on the image to be processed.

[0163] Example 5:

[0164] To achieve the above objectives, this application also provides a computer device 4, including: a processor 42 and a memory 41 communicatively connected to the processor 42; the memory stores computer-executed instructions;

[0165] The processor executes the computer execution instructions stored in memory 41 to implement the aforementioned training method and image processing method based on the image processing model. The components of the training device and image processing device based on the image processing model can be distributed across different computer devices. Computer device 4 can be a smartphone, tablet, laptop, desktop computer, rack server, blade server, tower server, or cabinet server (including standalone servers or server clusters composed of multiple application servers), etc. The computer device in this embodiment includes, but is not limited to, memory 41 and processor 42, which can be interconnected via a system bus. Figure 8 As shown. It should be noted that, Figure 8 Only computer devices with components are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead. In this embodiment, memory 41 (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, memory 41 may be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, memory 41 may also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Of course, memory 41 may also include both internal storage units and external storage devices of the computer device. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device, such as the program code of the image processing model-based training device and image processing device in Embodiments 3 and 4. Furthermore, the memory 41 can also be used to temporarily store various types of data that have been output or will be output. The processor 42 in some embodiments may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, for example, to run the image processing model-based training device and image processing device to implement the image processing model-based training method and image processing method of the above embodiments.

[0166] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application. It should be understood that the processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The memory may include high-speed RAM, and may also include non-volatile memory (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk, or optical disk, etc.

[0167] To achieve the above objectives, this application also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, App application store, etc., which stores computer-executable instructions. When the program is executed by the processor 42, it implements the corresponding function. The computer-readable storage medium of this embodiment is used to store computer-executable instructions for implementing the training method and image processing method based on the image processing model, and when executed by the processor 42, it implements the training method and image processing method based on the image processing model described above.

[0168] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0169] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0170] This application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described training method and image processing based on an image processing model.

[0171] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0172] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0173] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A training method based on an image processing model, characterized in that, include: Obtain the image to be trained; The image to be trained is input into a preset initial model. The image is then convolved using the first convolutional kernel of the convolutional layer in the initial model to obtain a first output vector. The image is then convolved using the second convolutional kernel of the convolutional layer in the initial model to obtain a second output vector. The first output vector represents the image processing result generated by machine learning on the image based on the first convolutional kernel. The second output vector represents the image processing result generated by machine learning on the image based on the second convolutional kernel. Based on the first output vector and the second output vector, a third convolution kernel is determined for the first convolution kernel and the second convolution kernel to obtain an image processing model with the third convolution kernel; wherein, the image processing model is used to process the image to be processed to obtain the image processing result of the image to be processed; The step of determining the third convolution kernel based on the first output vector and the second output vector includes: If it is determined that the difference between the first output vector and the training label of the image to be trained is less than a preset first training threshold, and the difference between the second output vector and the training label of the image to be trained is less than a preset second training threshold, then the first convolutional kernel and the second convolutional kernel are reparameterized to obtain the third convolutional kernel; wherein, the reparameterization is used to integrate the weights in at least one convolutional kernel to form a convolutional kernel; the first training threshold reflects that the first convolutional kernel that generates the first output vector based on the image to be trained is capable of image processing on the image to be trained and obtaining an image processing result matching the training label; the second training threshold reflects that the second convolutional kernel that generates the second output vector based on the image to be trained is capable of image processing on the image to be trained and obtaining an image processing result matching the training label.

2. The method according to claim 1, characterized in that, The second convolutional kernel based on the convolutional layer in the initial model performs convolution processing on the image to be trained to obtain a second output vector, including: The training image is convolved based on the occlusion kernel in the second convolution kernel to obtain occlusion output data. The occlusion output data is convolved based on the unit kernel corresponding to the occlusion kernel in the second convolution kernel to obtain the second output vector.

3. The method according to claim 1 or 2, characterized in that, Before inputting the image to be trained into the preset initial model, the method further includes: A second convolutional kernel corresponding to the first convolutional kernel is created in the convolutional layer of the initial model; wherein the second convolutional kernel is arranged in parallel with the first convolutional kernel; the second convolutional kernel includes an occlusion kernel and a unit kernel corresponding to the occlusion kernel, the occlusion kernel and the unit kernel are arranged in series; the occlusion kernel is used to perform convolution processing on the image to be trained to obtain an occlusion output vector; the unit kernel is used to perform convolution processing on the occlusion output vector to obtain a second output vector.

4. The method according to claim 1 or 2, characterized in that, Before obtaining the image processing model with the third convolutional kernel, the following steps are also included: The image to be inferred is obtained, and the image to be inferred is convolved based on the third convolution kernel of the convolutional layer in the initial model to obtain a third output vector; wherein, the third output vector is used to characterize the image processing result generated by image processing of the image to be inferred based on the third convolution kernel; If it is determined that the difference between the third output vector and the inference label of the image to be inferred is greater than a preset inference threshold, then the initial model is trained using the image to be inferred to obtain the image processing model.

5. An image processing method based on an image processing model, characterized in that, include: Obtain the image to be processed, input the image to be processed into a preset image processing model, perform convolution processing on the image to be processed based on the third convolution kernel of the convolutional layer in the image processing model, and obtain the image processing result corresponding to the image to be processed. The image processing model is the image processing model described in any one of claims 1-4.

6. A training device based on an image processing model, characterized in that, include: The training input module is used to acquire the image to be trained; The training module is used to input the image to be trained into a preset initial model, perform convolution processing on the image to be trained based on the first convolution kernel of the convolutional layer in the initial model to obtain a first output vector, and perform convolution processing on the image to be trained based on the second convolution kernel of the convolutional layer in the initial model to obtain a second output vector; wherein, the first output vector is used to represent the image processing result generated by machine learning on the image to be trained based on the first convolution kernel; the second output vector is used to represent the image processing result generated by machine learning on the image to be trained based on the second convolution kernel; The model generation module is used to determine a third convolution kernel based on the first output vector and the second convolution kernel, so as to obtain an image processing model with the third convolution kernel; wherein, the image processing model is used to process the image to be processed to obtain the image processing result of the image to be processed; The model generation module is specifically configured to perform reparameterization processing on the first convolutional kernel and the second convolutional kernel to obtain the third convolutional kernel if it is determined that the difference between the first output vector and the training label of the image to be trained is less than a preset first training threshold, and the difference between the second output vector and the training label of the image to be trained is less than a preset second training threshold; wherein, the reparameterization processing is used to integrate the weights in at least one convolutional kernel to form a convolutional kernel; the first training threshold reflects that the first convolutional kernel that generates the first output vector based on the image to be trained can perform image processing on the image to be trained and obtain an image processing result matching the training label; the second training threshold reflects that the second convolutional kernel that generates the second output vector based on the image to be trained can perform image processing on the image to be trained and obtain an image processing result matching the training label.

7. An image processing device based on an image processing model, characterized in that, include: The input processing module is used to acquire the image to be processed and input the image to be processed into a preset image processing model; An image processing module is used to perform convolution processing on the image to be processed based on the third convolution kernel of the convolution layer in the image processing model to obtain the image processing result corresponding to the image to be processed; wherein, the image processing model is the image processing model in the device of claim 6.

8. A computer device, characterized in that, include: A processor and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 5.

Images