Method for generating high-resolution image by using convolutional neural network model, optimization system and human-machine interface using convolutional neural network model

Abstract

A method for generating a high-resolution image using a convolutional neural network model, an optimization system and a human-machine interface using the convolutional neural network model, the method using the convolutional neural network model comprising: receiving a feature image by a standard convolutional layer; receiving, by a plurality of pruned convolutional layers, a first output of the standard convolutional layer and generating a second output; performing pruning processing on an input by using convolution kernels of different sizes by a multi-layer path residual connection block and generating a plurality of residual outputs, and combining the input and the plurality of residual outputs as an input of one of a plurality of pruned convolution layers, the multi-layer path residual connection block being connected between the plurality of pruned convolution layers; up-sampling, by an up-sampling processing layer, the second outputs of the plurality of pruned convolutional layers to produce a third output; and performing residual processing on the third output and the feature image by the first residual connection layer to generate a high-resolution image.

Description

Technical Field

[0001] This application relates to a method for generating high-resolution images using a convolutional neural network model, and more particularly to an optimized convolutional neural network model, a method for using the model, an optimization system for optimizing the model, a human-computer interface for using the optimized model, and a method for training the model. Background Technology

[0002] Generally, embedded visualization systems (such as human-machine interfaces) have limited hardware resources (such as ROM and RAM), making it difficult to display beautiful and detailed images. To improve the quality of the displayed images, high-quality images need to be stored directly in the system and read directly when needed for display. However, using this method, these high-quality images will occupy a large amount of system storage space, easily leading to insufficient available space.

[0003] Additionally, some technologies on the market allow human-machine interfaces to store only small-sized images and dynamically enlarge them to the required size when needed, through calculations such as bicubic interpolation. This can save a significant amount of internal storage space. However, using these technologies requires a certain level of computing resources, and due to the limited hardware capabilities of embedded systems, human-machine interfaces often struggle to achieve the required instruction cycles, resulting in less than satisfactory performance (e.g., noticeable delays in animation effects).

[0004] In light of this, some devices or platforms on the market use Convolutional Neural Network (CNN) models to perform dynamic image upscaling to save system storage space. More specifically, this technique first trains the CNN model using a large number of images, and then, when in use, inputs a small-sized (i.e., low-pixel) image into the trained CNN model for prediction, so that the CNN model can directly generate a large-sized (i.e., high-pixel) image.

[0005] However, CNN models typically use a large number of parameters, requiring the backend server hardware resources of the device or platform to be extremely powerful enough to handle the sheer volume of parameters. Therefore, it remains difficult to implement techniques using CNN models for dynamically upscaling input images into the human-machine interface of embedded systems. Summary of the Invention

[0006] The purpose of this application is to provide a method, optimization system, human-computer interface for using a convolutional neural network (CNN) model to generate high-resolution images, and a training method for the CNN model. By optimizing the CNN model, the number of parameters used by the CNN model can be reduced, thereby enabling embedded systems with limited hardware resources to use the CNN model to dynamically enlarge images.

[0007] In one embodiment, the convolutional neural network model of this application includes at least a standard convolutional layer, multiple pruning convolutional layers, an upsampling layer, a first residual connection layer, and a multi-layer path residual connection block connecting the multiple pruning convolutional layers. Furthermore, the method of generating high-resolution images using the convolutional neural network model of this application includes:

[0008] This standard convolutional layer receives a feature image as input;

[0009] The multiple pruned convolutional layers receive a first output of the standard convolutional layer as input and produce a second output;

[0010] The upsampling processing layer upsamples the second output to produce a third output;

[0011] The first residual connection layer combines the third output and the feature image to generate a processed feature image, wherein the resolution of the processed feature image is higher than the resolution of the feature image; and

[0012] The multi-path residual connection block receives the output of the first pruned convolutional layer among the plurality of pruned convolutional layers as a multi-path input. The multi-path residual connection block includes a first multi-path residual processing layer, a second multi-path residual processing layer, a third multi-path residual processing layer, and a second residual connection layer. The method includes:

[0013] The first multipath residual processing layer processes the multipath input and produces a first residual output by using a first sub-pruning layer with a first convolutional kernel;

[0014] The second multipath residual processing layer processes the multipath input and produces a second residual output by using a second sub-pruning layer with a second convolutional kernel, wherein the second convolutional kernel is different from the first convolutional kernel;

[0015] The third multipath residual processing layer processes the multipath input and produces a third residual output by using a third sub-pruning layer with a third convolutional kernel, wherein the third convolutional kernel is different from the first convolutional kernel and the second convolutional kernel; and

[0016] The second residual connection layer combines the first residual output, the second residual output, the third residual output, and the multipath input to generate a fourth residual output, wherein the fourth residual output is the input of the second pruned convolutional layer among the plurality of pruned convolutional layers.

[0017] In one embodiment, the convolutional neural network model optimization system of this application includes:

[0018] A receiving device is configured to receive the convolutional neural network model as described above;

[0019] A storage device configured to store multiple instructions; and

[0020] A processor is configured to execute the plurality of instructions to optimize the convolutional neural network model, wherein the plurality of instructions include optimizing the number of weight parameter values ​​of the convolutional neural network model using a plurality of pruning processes and at least one multi-path residual connection process.

[0021] In one embodiment, the human-machine interface of this application includes:

[0022] A receiving device is configured to receive an input image;

[0023] A storage device configured to store the convolutional neural network model as described above, wherein the convolutional neural network model includes at least a plurality of pruning processes for optimizing the number of weight parameter values ​​and at least one multi-path residual connection process; and

[0024] A processor is configured to infer from the input image using a convolutional neural network model and optimized weight parameter values ​​to produce an output image, wherein the size and resolution of the output image are larger than the size and resolution of the input image.

[0025] In one embodiment, the training method for the convolutional neural network model of this application includes:

[0026] a) Obtain the convolutional neural network model as described above;

[0027] b) Obtain a training dataset and a validation dataset;

[0028] c) Use the training dataset to train the convolutional neural network model, and use the validation dataset to validate the convolutional neural network model;

[0029] d) Repeat step c) before a preset number of training rounds is reached; and

[0030] e) After the preset number of training rounds is reached, end the training of the convolutional neural network model and record one or more weight parameter values ​​after training.

[0031] This application reduces the number of weight parameters in a convolutional neural network model through optimization, and then integrates the optimized convolutional neural network model and the trained weight parameters into a human-machine interface. Compared to related technologies, this application enables human-machine interfaces using embedded systems with limited hardware resources to dynamically upscale low-quality images using convolutional neural network models to produce high-quality images. Attached Figure Description

[0032] Figure 1 This is an embodiment of the block diagram of the optimization system of this application;

[0033] Figure 2 This is an embodiment of the block diagram of the human-machine interface of this application;

[0034] Figure 3 This is a first embodiment of the optimized convolutional neural network model of this application;

[0035] Figure 4 The second embodiment is a schematic diagram of the optimized convolutional neural network model of this application;

[0036] Figure 5 An example of a schematic diagram illustrating different convolutional kernels used in multi-layer path residual connection blocks;

[0037] Figure 6 This is an embodiment of the model training flowchart of this application;

[0038] Figure 7 An example of using a flowchart for the model of this application;

[0039] Figure 8 This is an embodiment of the model training flowchart of this application;

[0040] Figure 9 This is a first embodiment of the flowchart for using the convolutional neural network model of this application.

[0041] Explanation of icon numbers

[0042] 1: Optimize the system

[0043] 11: First Processor

[0044] 12: First receiving device

[0045] 13: First storage device

[0046] 2: CNN model

[0047] 3: Optimize the CNN model

[0048] 31: Color Gamut Conversion Layer

[0049] 311: First color gamut conversion layer

[0050] 312: Second color gamut conversion layer

[0051] 32: Standard Convolutional Layer

[0052] 33: Pruned convolutional layer

[0053] 331: First-pointing pruned convolution

[0054] 332: First-depth pruned convolution

[0055] 333: Second-pointing pruned convolution

[0056] 334: Second-depth pruned convolution

[0057] 335: Third-pointing pruning convolution

[0058] 34: Multi-layer path residual connection block

[0059] 341: First Multipath Residual Processing Layer

[0060] 3411, 3421, 3431: First dispersion depth pruned convolution

[0061] 3412, 3422, 3432: First Dispersed Pointing Pruned Convolution

[0062] 3413, 3423, 3433: Activation function layers

[0063] 3414, 3424, 3434: Second dispersion depth pruned convolution

[0064] 3415, 3425, 3435: Second Dispersion-Directional Pruned Convolution

[0065] 342: Second Multipath Residual Processing Layer

[0066] 343: Third Multipath Residual Processing Layer

[0067] 344: Second Residual Connector Layer

[0068] 35: Upsampling Processing Layer

[0069] 36: First Residual Connectivity Layer

[0070] 37: First activation function layer

[0071] 38: Second activation function layer

[0072] 4: Human-machine interface

[0073] 41: Second Processor

[0074] 42: Second receiving device

[0075] 43: Second storage device

[0076] 5: Input Image

[0077] 6: Output Image

[0078] 71: First Filter

[0079] 72: Second Filter

[0080] 73: Third Filter

[0081] Yi: Multipath Input

[0082] Y1: First residual output

[0083] Y2: Second residual output

[0084] Y3: Third residual output

[0085] Y0: Fourth residual output

[0086] S81~S85: Training Steps

[0087] S91~S94, S921~S925: Usage Steps Detailed Implementation

[0088] This application discloses a method for generating high-resolution images using a Convolutional Neural Network (CNN) model, an optimization system for optimizing the CNN model, and a human-computer interface using the optimized CNN model. Specifically, this application first trains a CNN model using the optimization system, and then optimizes the CNN model through pruning and multi-path residual connection processing to produce an optimized CNN model with a significantly reduced number of parameters. For human-computer interfaces using embedded systems, because the optimized CNN model has fewer parameters, the human-computer interface, despite having extremely limited hardware resources, can still use the optimized CNN model to dynamically upscale low-quality input images to obtain high-quality output images. In this way, the human-computer interface only needs to store low-quality images to generate high-quality images through the inference of the optimized CNN model, thereby saving internal storage space while obtaining high-quality output that meets user needs.

[0089] See Figure 1 This is an embodiment of the block diagram of the optimization system of this application. Figure 1This application discloses a CNN model optimization system (hereinafter referred to as optimization system 1). Optimization system 1 receives or builds a CNN model 2, and performs the aforementioned optimization processing on the CNN model 2 to generate an optimized CNN model 3. In one embodiment, the user can import the optimized CNN model 3 into the human-machine interface (e.g., using an embedded system) of the human-machine interface. Figure 2 In the human-machine interface 4 shown. Since the optimized CNN model 3 has fewer parameters, even when using an embedded system, the hardware performance of the human-machine interface 4 is still sufficient to perform calculations using the optimized CNN model 3. More specifically, in addition to generating the optimized CNN model 3, the optimization system 1 also needs to train the optimized CNN model 3 to generate trained weight parameter values ​​(described in detail later). In this embodiment, in addition to storing the optimized CNN model 3, the human-machine interface 4 also needs to input the trained weight parameter values ​​into the optimized CNN model 3, thereby dynamically generating and outputting high-quality images according to the user's needs.

[0090] like Figure 1 As shown, the optimization system 1 includes a first processor 11, a first receiving device 12, and a first storage device 13. In one embodiment, the optimization system 1 is an electronic device used to build and train a model, such as a personal computer, industrial computer, ground server, cloud server, or other electronic device with high hardware performance sufficient to train a neural network model. The first processor 11 can be a central processing unit (CPU), microcontroller unit (MCU), programmable logic controller (PLC), system on chip (SoC), or field programmable gate array (FPGA). The first receiving device 12 can be a wired transmission device (e.g., a USB connector or RS-232 connector) or a wireless transmission device (e.g., an infrared transmission device, Bluetooth transmission device, Wi-Fi transmission device, or 4G / 5G transmission device). The first storage device 13 can be a hard disk, solid-state drive, or various types of memory. However, the above are only some specific embodiments of this application, and are not limited to the above.

[0091] The first receiving device 12 is used to receive a CNN model 2. The CNN model 2 may be, for example, a standard CNN model that has not yet been fine-tuned or optimized. In another embodiment, the user may also directly create a new CNN model 2 by operating the optimization system 1, but this is not a limitation. The first storage device 13 stores a plurality of instructions, which are relevant program instructions required for building, training, and optimizing the model. The first processor 11 is used to execute the plurality of instructions in the first storage device 13 to optimize the CNN model 2 and generate an optimized CNN model 3.

[0092] Specifically, after executing the instructions, the first processor 11 can perform pruning and multi-path residual connection processing on the CNN model 2 (detailed later) to reduce the number of weight parameter values ​​used by the CNN model 2, thereby generating an optimized CNN model 3 with fewer weight parameter values ​​but unchanged (or even better) performance. In other words, the optimized CNN model 3 has fewer weight parameter values ​​than the CNN model 2.

[0093] After generating the optimized CNN model 3, the optimization system 1 then uses the training data to train the optimized CNN model 3 (details to follow).

[0094] Please see below. Figure 2 This is an embodiment of the block diagram of the human-machine interface of this application. Figure 2 The present application discloses a human-machine interface 4, which includes at least a second processor 41, a second receiving device 42, a second storage device 43, and a display device (not shown). The second processor 41, the second receiving device 42, and the second storage device 43 are the same as or similar to the aforementioned first processor 11, first receiving device 12, and first storage device 13, and will not be described again here. In this embodiment, the hardware performance of the human-machine interface 4 (e.g., the second processor 41, the second receiving device 42, and the second storage device 43) is inferior to that of the optimized system 1, and the human-machine interface 4 uses an embedded system to compensate for the inferior hardware performance.

[0095] At Figure 2 In one embodiment, the human-machine interface 4 stores the optimized CNN model 3 generated by the optimization system 1 through the second storage device 43. As described above, compared with the standard CNN model 2, the optimized CNN model 3 also includes pruning processing to optimize the number of weight parameter values ​​and multi-path residual connection processing. In one embodiment, the human-machine interface 4 stores the optimized CNN model 3, and after the user imports the trained weight parameter values ​​into the human-machine interface 4, the human-machine interface 4 can combine the optimized CNN model 3 and the trained weight parameter values ​​to perform dynamic upscaling processing on the input low-quality image to generate and output a high-quality image.

[0096] The second receiving device 42 is used to receive the input image 5. In one embodiment, the human-machine interface 4 receives the low-quality input image 5 from the outside through the second receiving device 42. In another embodiment, the second receiving device 42 is connected to the second storage device 43, and the human-machine interface 4 reads the low-quality input image 5 pre-stored in the second storage device 43 through the second receiving device 42. In this application, after determining the input image 5 and the output requirements (e.g., magnification or required resolution), the second processor 41 uses the optimized CNN model 3 combined with the trained weight parameter values ​​to infer the input image 5 to generate the output image 6. Through the operation of the optimized CNN model 3, the size and resolution of the output image 6 can be larger than the size and resolution of the input image 5, and the output image 6 can meet the output requirements.

[0097] By integrating the optimized CNN model 3 onto the human-machine interface 4, the interface can utilize limited hardware resources to execute the optimized CNN model 3 for image inference, achieving dynamic upscaling of the input image 5. Thus, while saving limited storage space (only storing the low-quality input image 5), the interface can still provide a high-quality output image 6 that meets the user's needs at any time. Furthermore, because the optimized CNN model 3 has fewer parameters, the latency of the interface can be minimized when executing the optimized CNN model 3.

[0098] See Figure 3 This is a first embodiment of the optimized CNN model of this application. Figure 3 As shown, the optimized CNN model 3 of this application includes at least a standard convolutional layer 32, multiple pruned convolutional layers 33, an upsampling processing layer 35, a first residual connection layer 36, and a multi-path residual connection block 34, wherein the multi-path residual connection block 34 is connected between multiple pruned convolutional layers 33, and has at least a first multi-path residual processing layer 341, a second multi-path residual processing layer 342, a third multi-path residual processing layer 343, and a second residual connection layer 344.

[0099] The standard convolutional layer 32 receives the input image 5 and performs standard convolution processing on it. It's worth noting that optimizing the CNN model 3 requires using a color gamut conversion layer 31 for color gamut conversion. The color gamut conversion layer 31 converts the input image 5, described in RGB, into a feature image described in a color gamut (e.g., Y, Cb, Cr), and the standard convolutional layer 32 performs standard convolution processing on this feature image. This standard convolution processing is common knowledge in the field of neural networks and will not be elaborated upon here. The color gamut refers to a color space, where Y represents luminance, Cb represents the blue-difference chroma component, and Cr represents the red-difference chroma component. Color gamut conversion is common knowledge in image processing and will not be elaborated upon here.

[0100] Multiple pruned convolutional layers 33 receive the first output of the standard convolutional layer and, after multiple pruning processes, produce a second output. Specifically, the pruning process is a model compression technique in convolutional neural networks, used to remove redundant and unimportant parameters and neurons from the CNN model, thereby reducing the model size and accelerating the model's inference process. This pruning process is common knowledge in the field of neural networks and will not be elaborated upon here.

[0101] The upsampling processing layer 35 receives the second outputs of multiple pruned convolutional layers 33 and, after upsampling processing, generates a third output. Specifically, the upsampling processing is a technique to improve image resolution by rearranging the pixels in the image, and this upsampling processing is common knowledge in the field of neural networks, and will not be elaborated further here.

[0102] The first residual connection layer 36 combines the third output of the upsampling processing layer 35 with the feature image generated by the color gamut conversion layer 31 to produce a processed feature image. Specifically, the residual connection uses the output of a previous layer in the neural network as the input of a subsequent layer, and allows the neural network to calculate and learn the difference between the input and output (i.e., the residual), thereby solving the gradient explosion problem that occurs as the depth of the neural network increases. Using a single layer of residual processing in a neural network is common knowledge in the field of neural networks and will not be elaborated further here.

[0103] Through the inference operations of the optimized CNN model 3, the resolution of the processed feature image will be higher than that of the input image. In one embodiment, another color gamut conversion layer 31 is connected after the first residual connection layer 36. This color gamut conversion layer 31 converts the processed feature image described in color gamut (e.g., Y, Cb, Cr) into an output image 6 described in RGB. In other words, the resolution of the output image 6 will be higher than that of the input image 5. If the human-machine interface 4 is equipped with the optimized CNN model 3, then when the human-machine interface 4 uses the optimized CNN model 3 to perform operations on the input image 5, it can generate the output image 6 and display the output image 6 on its screen (not shown in the figure).

[0104] The multi-path residual connection block 34 connects multiple pruned convolutional layers. In one embodiment, the multi-path residual connection block 34 connects between a first pruned convolutional layer and a second pruned convolutional layer. In other words, the output of the first pruned convolutional layer is the input of the multi-path residual connection block 34, and the output of the multi-path residual connection block 34 is the input of the second pruned convolutional layer.

[0105] At Figure 3 In the embodiment, the multi-path residual connection block 34 uses the output of the first pruned convolutional layer as the multi-path input Yi, and the first multi-path residual processing layer 341, the second multi-path residual processing layer 342 and the third multi-path residual processing layer 343 in the multi-path residual connection block 34 all use the same multi-path input Yi as input.

[0106] The first multipath residual processing layer 341 includes at least a first sub-pruning layer using a first convolutional kernel, and processes the multipath input Yi through the first sub-pruning layer to generate a first residual output Y1. The second multipath residual processing layer 342 includes at least a second sub-pruning layer using a second convolutional kernel, and processes the multipath input Yi through the second sub-pruning layer to generate a second residual output Y2. The third multipath residual processing layer 343 includes at least a third sub-pruning layer using a third convolutional kernel, and processes the multipath input Yi through the third sub-pruning layer to generate a third residual output Y3. In the technical solution of this application, the third convolutional kernel is different from the second convolutional kernel, and the second convolutional kernel is different from the first convolutional kernel. Therefore, the residual outputs of these three layers are residually concatenated to reduce the occurrence of gradient explosion.

[0107] The second residual connection layer 344 simultaneously connects the first multipath residual processing layer 341, the second multipath residual processing layer 342, and the third multipath residual processing layer 343, to combine the first residual output Y1, the second residual output Y2, the third residual output Y3, and the multipath input Yi, and generate a fourth residual output Yo. In this embodiment, the second pruned convolutional layer among the multiple pruned convolutional layers 33 uses the fourth residual output Yo as its input and performs further pruning processing on this input.

[0108] Please also refer to Figure 9 This is a first embodiment of the flowchart for using the CNN model of this application. For ease of understanding, the following is combined with... Figure 3 and Figure 9 This paper explains how to use the optimized CNN model 3 of this application to transform a low-resolution input image 5 into a high-resolution output image 6.

[0109] Figure 9 The method shown is mainly applied to Figure 3 The optimized CNN model 3 is shown. Specifically, firstly, the optimized CNN model 3 receives the feature image extracted from the input image 5 as input by the standard convolutional layer 32, and the standard convolutional layer 32 performs standard convolution processing on the feature image to produce a first output. Next, the optimized CNN model 3 receives the first output of the standard convolutional layer 32 as input through multiple pruned convolutional layers 33, and produces a second output after pruning processing (step S92). It is worth mentioning that before the multiple pruned convolutional layers 33 produce the second output, the optimized CNN model 3 further performs the following operation through multi-layer path residual connection blocks 34.

[0110] Specifically, in step S92, the CNN model 3 is optimized through the first pruned convolutional layer (e.g., ...) of the multiple pruned convolutional layers 33. Figure 4The first pointed-prune convolution 331 (shown) prunes the first output and produces an output, while the multi-path residual connection block receives the output of the first pruned convolution layer as the multipath input Yi (step S921). Next, the first multipath residual processing layer 341 in the multi-path residual connection block 34 processes the multipath input Yi using the first sub-pruning layer of the first convolution kernel and produces the first residual output Y1 (step S922). The second multipath residual processing layer 342 in the multi-path residual connection block 34 processes the multipath input Yi using the second sub-pruning layer of the second convolution kernel and produces the second residual output Y2 (step S923). The third multipath residual processing layer 343 in the multi-path residual connection block 34 processes the multipath input Yi using the third sub-pruning layer of the third convolution kernel and produces the third residual output Y3 (step S924). The first, second, and third convolution kernels are distinct from each other. It is worth mentioning that steps S922 to S924 do not have an execution order. The multi-layer path residual connection block 34 can execute steps S922 to S924 simultaneously based on multi-task processing, instead of... Figure 9 The examples shown are limited to those shown.

[0111] Next, the optimized CNN model 3 further combines the multi-path input Yi, the first residual output Y1, the second residual output Y2, and the third residual output Y3 through the second residual connection layer 344 in the multi-path residual connection block 34 to generate the fourth residual output Yo (step S925). Thus, the optimized CNN model 3 uses the fourth residual output Yo as the second pruned convolutional layer in the multiple pruned convolutional layers 33 (e.g., Figure 4 The input of the first depth pruning convolutional layer 332 shown allows multiple pruning convolutional layers 33 to then perform pruning processing based on this input, ultimately producing the second output.

[0112] Next, the optimized CNN model 3 receives the second output from multiple pruned convolutional layers 33 via an upsampling processing layer 35, and upsamples the second output to generate a third output (step S93). Finally, the optimized CNN model 3 combines the third output with the feature image via a first residual connection layer 36 to generate a processed feature image (step S94). In this application, the processed feature image is a high-resolution image (i.e., the resolution of the processed feature image is higher than that of the input image 5). After color gamut conversion, the optimized CNN model 3 can output even more... Figure 3 The high-resolution output image shown is 6.

[0113] Please also refer to Figure 4 This is a second embodiment of the optimized CNN model of this application. Figure 4 More clearly revealed Figure 3 The specific structure of the optimized CNN model 3 is shown.

[0114] As described above, the optimized CNN model 3 of this application includes multiple pruned convolutional layers 33, and as... Figure 4 As shown, the multiple pruned convolutional layers 33 include multiple pointwise pruning convolution (PW Conv) layers 331, 333, and 335, and multiple depthwise pruning convolution (DW Conv) layers 332 and 334. More specifically, the multiple pruned convolutional layers 33 sequentially include a first pointwise pruning convolutional layer 331, a first depthwise pruning convolutional layer 332, a second pointwise pruning convolutional layer 333, a second depthwise pruning convolutional layer 334, and a third pointwise pruning convolutional layer 335.

[0115] The first pointed-pruned convolutional layer 331 receives the first output of the standard convolutional layer 32 as input and generates a first pruned output after pruning. In this embodiment, the multi-path residual connection block 34 uses the first pruned output as its multi-path input Yi. The first depth-pruned convolutional layer 332 receives the fourth residual output Yo of the multi-path residual connection block 34 as input and generates a second pruned output after pruning.

[0116] The second pointed-pruned convolutional layer 333 receives the second pruned output of the first depth-pruned convolutional layer 332 as input, and generates a third pruned output after pruning. The second depth-pruned convolutional layer 334 receives the third pruned output of the second pointed-pruned convolutional layer 333 as input, and generates a fourth pruned output after pruning. The third pointed-pruned convolutional layer 335 receives the fourth pruned output of the second depth-pruned convolutional layer 334 as input, and generates the second output after pruning. Finally, the upsampling layer 35 takes the second output of the second depth-pruned convolutional layer 334 as input, and generates the third output after upsampling.

[0117] As described above, the first multipath residual processing layer 36 receives the third output of the upsampling processing layer 35 and the feature image, and generates the processed feature image.

[0118] At Figure 4 In this embodiment, the standard convolutional layer 32 uses a 3x3 kernel for standard convolution processing. The upsampling layer is a pixel shuffle layer.

[0119] The optimized CNN model 3 of this application also has multiple activation function layers (the first activation function layer 37 and the second activation function layer 38 are taken as examples below). Activation functions introduce nonlinear factors to enable the neural network to learn nonlinear images, thereby improving the model's learning ability and controlling the output range. Specifically, activation functions are common knowledge in the field of neural networks and will not be elaborated here.

[0120] like Figure 4 As shown, a first activation function layer 37 is connected between the standard convolutional layer 32 and the first pruned convolutional layer 331; a first activation function layer 37 is connected between the first pruned convolutional layer 331 and the multi-level path residual connection block 34; a first activation function layer 37 is connected between the multi-level path residual connection block 34 and the first depth pruned convolutional layer 332; a first activation function layer 37 is connected between the second pruned convolutional layer 333 and the second depth pruned convolutional layer 334; and a second activation function layer 38 is connected after the first residual connection layer 36.

[0121] In one embodiment, the first activation function layer 37 is a parametric rectified linear unit (PReLu) layer, and the second activation function layer is a rectified linear unit (ReLu) layer.

[0122] As mentioned above, the multi-path residual connection block 34 includes a first multi-path residual processing layer 341, a second multi-path residual processing layer 342, and a third multi-path residual processing layer 343 in parallel, and the outputs of the first multi-path residual processing layer 341, the second multi-path residual processing layer 342, and the third multi-path residual processing layer 343 are jointly connected to the second residual connection layer 344.

[0123] The first multipath residual processing layer 341 sequentially includes a first distributed depth-pruned convolution (DW Conv) layer 3411, a first distributed directional pruned convolution (PW Conv) layer 3412, a second distributed depth-pruned convolution (DW Conv) layer 3414, and a second distributed directional pruned convolution (PW Conv) layer 3415. The second multipath residual processing layer 342 sequentially includes a first distributed depth-pruned convolution (DW Conv) layer 3421, a first distributed directional pruned convolution (PW Conv) layer 3422, a second distributed depth-pruned convolution (DW Conv) layer 3424, and a second distributed directional pruned convolution (PW Conv) layer 3425. The third multipath residual processing layer 343 sequentially includes a first dispersed depth pruning convolution (DW Conv) layer 3431, a first dispersed pointing pruning convolution (PW Conv) layer 3432, a second dispersed depth pruning convolution (DW Conv) layer 3434, and a second dispersed pointing pruning convolution (PW Conv) layer 3435.

[0124] The first divergent depth-pruning convolutional layers 3411, 3421, and 3431 of the first multipath residual processing layer 341, the second multipath residual processing layer 342, and the third multipath residual processing layer 343 all receive multipath input Yi and perform pruning processing respectively. The second divergent pointing-pruning convolutional layers 3415, 3425, and 3435 of the first multipath residual processing layer 3411, the second multipath residual processing layer 3421, and the third multipath residual processing layer 3431 generate the first residual output Y1, the second residual output Y2, and the third residual output Y3 respectively. The second residual connection layer 344 combines the first residual output Y1, the second residual output Y2, the third residual output Y3, and the multipath input Yi to generate the fourth residual output Yo.

[0125] like Figure 4 As shown, the first multipath residual processing layer 341, the second multipath residual processing layer 342, and the third multipath residual processing layer 343 have similar structures. The difference lies in the convolution kernels used in the first dispersion depth pruning convolutional layers 3411, 3421, and 3431 of each of the three multipath residual processing layers. In one embodiment, the first dispersion depth pruning convolutional layer 3411 of the first multipath residual processing layer 341 is a first sub-pruning layer using a first convolutional kernel of size 3x3; the first dispersion depth pruning convolutional layer 3421 of the second multipath residual processing layer 342 is a second sub-pruning layer using a second convolutional kernel of size 1x3; and the first dispersion depth pruning convolutional layer 3431 of the third multipath residual processing layer 343 is a third sub-pruning layer using a third convolutional kernel of size 3x1.

[0126] It is worth mentioning that the multi-layer path residual connection block 34 in this application is an improvement based on the ResNeXt architecture proposed by Google. It splits a single path using a 3x3 convolutional kernel into three paths using 3x3, 1x3, and 3x1 convolutional kernels respectively. Combined with residual processing, this reduces the number of model parameters while improving the perceived field of view of the output image. Experimental results show that the CNN model optimized using the multi-layer path residual connection block 34 in this application reduces the number of parameters by at least 31% compared to the CNN model using the ResNeXt architecture.

[0127] Please also refer to Figure 4 and Figure 5 ,in Figure 5 This is an example of a schematic diagram illustrating different convolutional kernels used in multi-layer path residual connection blocks. For example... Figure 5 As shown, using a 3x3 convolution kernel means using a 3x3 matrix as a filter (e.g., the first filter 71) to convolve the input image, thereby extracting features from the input image based on a 3x3 range. Using a 3x1 convolution kernel means using a 3x1 matrix as a filter (e.g., the second filter 72) to convolve the input image, thereby extracting features from the input image based on a 3x1 range. Using a 1x3 convolution kernel means using a 1x3 matrix as a filter (e.g., the third filter 73) to convolve the input image, thereby extracting features from the input image based on a 1x3 range. The technical solution of this application is to simultaneously perform three pruning processes on the multi-path input Yi, based on a 3x3 first convolution kernel, a 3x1 second convolution kernel, and a 1x3 third convolution kernel, and then sum the results of the three pruning processes. This application performs pruning on the same input (i.e., multi-path input Yi) on multiple parallel paths, and finally adds the output results of all paths and the multi-path input Yi together. This can improve the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM index) of the output image 6 compared to the input image 5, which can effectively improve the detail of the output image 6.

[0128] It is worth mentioning that, such as Figure 4As shown, the first multipath residual processing layer 341, the second multipath residual processing layer 342, and the third multipath residual processing layer 343 in the multi-path residual connection block 34 respectively have activation function layers 3413, 3423, and 3433. These activation function layers 3413, 3423, and 3433 are respectively disposed between the first dispersed pointing pruning convolutional layers 3412, 3422, and 3432 and the second dispersed depth pruning convolutional layers 3414, 3424, and 3434. In one embodiment, the activation function layers 3413, 3423, and 3433 are PReLu layers.

[0129] The optimized CNN model 3 of this application processes the input image 5 through multiple directional pruning layers, multiple depth pruning layers, and multi-path residual processing layers. This significantly reduces the number of parameters while improving the image detail of the output image 6 generated after model inference. Therefore, it is not only suitable for embedded systems with limited hardware resources, but also does not degrade image quality or cause image processing delays.

[0130] Please see below. Figure 6 This is an example of a model training flowchart from this application. Figure 6 As shown, when training the optimized CNN model 3 of this application, the user first needs to prepare a large number of training images for the required image types and extract the features of these images. Then, incremental image processing is used to increase the number of training images, thereby reducing the time and difficulty of collecting training images. Next, the user can use a computer device (e.g., optimization system 1) to build a standard CNN model (e.g., Figure 1 The CNN model shown is 2), and the CNN model is optimized using the above-mentioned techniques to generate an optimized CNN model 3. In this application, a computer device can optimize the CNN model by executing instructions to establish pruning processing and multi-path residual connection processing in the CNN model.

[0131] After the optimized CNN model 3 is built, the user can manually or automatically use a computer device to divide the training images into a training dataset for training the model and a validation dataset for validating the model. Generally, the training dataset accounts for 80% of all training images, and the validation dataset accounts for 20%, but this is not a limitation. The computer device repeatedly trains the optimized CNN model 3 using the training and validation datasets (e.g., at least 10,000 training cycles) to obtain the trained weight parameter values. After training, the user can import the optimized CNN model 3 and the trained weight parameter values ​​into the human-computer interface 4. In this way, the human-computer interface 4 can use the optimized CNN model 3 combined with the trained weight parameter values ​​to infer from low-quality input images to produce high-quality output images.

[0132] Please see Figure 7 This is an example of a flowchart used in this application. Figure 7 The flowchart is primarily applicable to the human-computer interface 4 with an optimized CNN model 3. In use, the user inputs a low-quality image into the human-computer interface 4 and specifies the magnification factor or required resolution, such as 2x, 4x, 8x magnification, or a resolution of 640x640 or 1024x1024. Next, the human-computer interface 4 uses the optimized CNN model 3, combined with trained weight parameter values, to process the input image (or a feature image after color gamut conversion) to produce the magnified image.

[0133] In one embodiment, the human-machine interface 4 can display the magnified image on a screen for direct viewing by the user. In other embodiments, the human-machine interface 4 can send the magnified image to the user via a transmission unit (not shown). Thus, the human-machine interface 4 only needs to receive or store images with relatively low image quality and small file size to perform dynamic magnification processing to obtain high-quality images. Furthermore, since the human-machine interface 4 uses an optimized CNN model 3 with fewer parameters, there is no processing delay.

[0134] Please see Figure 8 This is an embodiment of the model training flowchart of this application. Figure 8 This describes the specific steps of the training method used in this application to train the optimized CNN model 3, and the training method can be applied to any computer device without limitation.

[0135] like Figure 8 As shown, when training the optimized CNN model 3, firstly, the computer device to perform the training procedure obtains the optimized CNN model 3 (step S81), and the computer device simultaneously obtains as follows: Figure 6 The training dataset and validation dataset are shown (step S82). As mentioned above, the computer device mainly automatically or manually distinguishes the imported training images (or image features) into a training dataset (approximately 80%) and a validation dataset (approximately 20%). Next, the computer device can use the training dataset to train the optimized CNN model 3, and simultaneously use the validation dataset to validate the optimized CNN model 3 (step S83).

[0136] Specifically, in each training round, the computer device trains the optimized CNN model 3 using the training dataset and validates it using the validation dataset. During training, the computer device continuously checks whether the current training round has reached the preset number of training rounds (step S84). If the current training round has not reached the preset number of training rounds (e.g., 10,000 times), the computer device repeats step S83 to perform the next round of training on the optimized CNN model 3 using the same training and validation datasets. When the current training round reaches the preset number of training rounds, the computer device ends the current training program and records the trained weight parameter values ​​from the trained optimized CNN model 3. As described above, the trained weight parameter values ​​can be output to the human-computer interface 4, allowing the human-computer interface 4 to use the optimized CNN model 3 combined with the trained weight parameter values ​​to infer from low-quality input images to produce high-quality output images.

Claims

1. A method for generating high-resolution images using a convolutional neural network model, said convolutional neural network model comprising a standard convolutional layer, multiple pruned convolutional layers, an upsampling layer, a first residual connection layer, and multi-layer path residual connection blocks connecting the multiple pruned convolutional layers, the method comprising: The standard convolutional layer receives the feature image as input; The plurality of pruned convolutional layers receive the first output of the standard convolutional layer as input and generate a second output; The upsampling processing layer upsamples the second output to generate a third output; The first residual connection layer combines the third output and the feature image to generate a processed feature image, wherein the resolution of the processed feature image is higher than the resolution of the feature image; and The multi-path residual connection block receives the output of the first pruned convolutional layer among the plurality of pruned convolutional layers as a multi-path input, wherein the multi-path residual connection block includes a first multi-path residual processing layer, a second multi-path residual processing layer, a third multi-path residual processing layer, and a second residual connection layer, and the method further includes: The first multipath residual processing layer processes the multipath input and generates a first residual output by using a first sub-pruning layer with a first convolutional kernel; The second multipath residual processing layer processes the multipath input and produces a second residual output by using a second sub-pruning layer with a second convolutional kernel, wherein the second convolutional kernel is different from the first convolutional kernel; The third multipath residual processing layer processes the multipath input and generates a third residual output by using a third sub-pruning layer of a third convolutional kernel, wherein the third convolutional kernel is different from the first convolutional kernel and the second convolutional kernel; and The second residual connection layer combines the first residual output, the second residual output, the third residual output, and the multipath input to generate a fourth residual output, wherein the fourth residual output is the input of the second pruned convolutional layer among the plurality of pruned convolutional layers.

2. The method according to claim 1, wherein the plurality of pruned convolutional layers includes a plurality of depth-pruned convolutional layers and a plurality of directional-pruned convolutional layers.

3. The method according to claim 1, wherein the plurality of pruning convolutional layers includes a first directional pruning convolutional layer, a first depth pruning convolutional layer, a second directional pruning convolutional layer, a second depth pruning convolutional layer, and a third directional pruning convolutional layer, and the method includes: The first pruning convolutional layer receives the first output as input and generates a first pruned output, wherein the first pruned output is the same as the multipath input; The first depth-pruned convolutional layer receives the fourth residual output as input and generates the second pruned output. The second pruning convolutional layer receives the second pruned output as input and generates the third pruned output; The second depth-pruned convolutional layer receives the third pruned output as input and generates the fourth pruned output; and The third pruning convolutional layer receives the fourth pruning output as input and produces the second output.

4. The method according to claim 1, wherein the upsampling processing layer is a pixel rearrangement layer.

5. The method of claim 1, wherein the standard convolutional layer uses a convolutional kernel of size 3x3.

6. The method according to claim 1, wherein the convolutional neural network model further comprises a plurality of activation function layers, the plurality of activation function layers being respectively disposed between the first pruned convolutional layer and the multi-layer path residual connection block, and between the multi-layer path residual connection block and the second pruned convolutional layer.

7. The method according to claim 1, wherein the first multipath residual processing layer, the second multipath residual processing layer, and the third multipath residual processing layer each comprise a first divergent depth pruning convolutional layer, a first divergent pointing convolutional layer, a second divergent depth pruning convolutional layer, and a second divergent pointing convolutional layer, wherein the first divergent depth pruning convolutional layer of the first multipath residual processing layer, the second multipath residual processing layer, and the third multipath residual processing layer receives the multipath input, and the second divergent pointing convolutional layer of the first multipath residual processing layer, the second multipath residual processing layer, and the third multipath residual processing layer respectively generates the first residual output, the second residual output, and the third residual output.

8. The method of claim 7, wherein the first dispersion depth pruning convolutional layer of the first multipath residual processing layer is a first sub-pruning layer using a first convolutional kernel of size 3x3, the first dispersion depth pruning convolutional layer of the second multipath residual processing layer is a second sub-pruning layer using a second convolutional kernel of size 1x3, and the first dispersion depth pruning convolutional layer of the third multipath residual processing layer is a third sub-pruning layer using the third convolutional kernel of size 3x1.

9. The method according to claim 3, wherein the convolutional neural network model further comprises a plurality of first activation function layers, the plurality of first activation function layers being respectively disposed between the standard convolutional layer and the first directional pruning convolutional layer, between the first directional pruning convolutional layer and the multi-layer path residual connection block, between the multi-layer path residual connection block and the first depth pruning convolutional layer, and between the second directional pruning convolutional layer and the second depth pruning convolutional layer.

10. The method of claim 9, wherein the convolutional neural network model further comprises a second activation function layer disposed after the first residual connection layer.

11. The method according to claim 10, wherein the plurality of first activation function layers are parameterized rectified linear unit layers, and the second activation function layer is a rectified linear unit layer.

12. The method according to claim 8, wherein activation function layers are respectively provided between the first scatter pointing pruning convolutional layer and the second scatter depth pruning convolutional layer of the first multipath residual processing layer, the second multipath residual processing layer and the third multipath residual processing layer.

13. The method according to claim 12, wherein the activation function layer is a parameterized rectified linear unit layer.

14. A convolutional neural network model optimization system, comprising: The receiving device is configured to receive a standard convolutional neural network model; A storage device configured to store multiple instructions; and A processor is configured to execute the plurality of instructions to optimize the standard convolutional neural network model to produce the convolutional neural network model of claim 1, wherein the plurality of instructions include optimizing the number of weight parameter values ​​of the convolutional neural network model using a plurality of pruning processes and at least one multi-path residual connection process.

15. A human-machine interface, comprising: A receiving device is configured to receive an input image; A storage device is configured to store the convolutional neural network model as claimed in claim 1, wherein the convolutional neural network model includes at least a plurality of pruning processes for optimizing the number of weight parameter values ​​and at least one multi-path residual connection process. and A processor is configured to infer from the input image using the convolutional neural network model and the trained weight parameter values ​​to produce an output image, wherein the size and resolution of the output image are greater than the size and resolution of the input image.

Images