Convolution operation circuit and related convolution operation method

Abstract

The invention discloses a convolution operation circuit and a related convolution operation method. The convolution operation circuit comprises a rotation judging unit, a convolution kernel adjusting unit and a convolution operation unit. The rotation judgment unit is used for determining a rotation state of input data. The convolution kernel adjusting unit is coupled to the rotation judging unit and is used for selectively adjusting an initial convolution kernel according to the rotation state so as to obtain an adjusted convolution kernel. The convolution operation unit is coupled to the convolution kernel adjustment unit and is used for performing convolution operation according to the adjusted convolution kernel and the input data so as to obtain a feature map.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a convolution operation circuit and a related convolution operation method that can be applied to a convolution-related layer of a deep learning model. Background Art

[0002] Deep learning technology is now widely used in various fields, such as image recognition, speech processing, and natural language processing. Deep learning models are typically composed of multi-layered neural networks capable of learning complex features and patterns from large amounts of data. Among these, convolutional neural networks (CNNs) have particularly distinguished themselves in image recognition due to their exceptional image processing capabilities. When performing image recognition tasks, CNNs effectively extract image features through convolution operations, which are then used for classification and other high-level processing. However, in real-world applications, the angle of the image acquisition device may cause the image to appear rotated, which can further affect the recognition accuracy of CNN models. This is because CNN models are typically trained on images with a fixed orientation and are not robust to rotated images. Existing approaches typically address this issue in two ways. One is to expand the training dataset. Specifically, by adding images with different rotation angles to the training dataset, the model's ability to recognize rotated images can be enhanced. However, while this approach can improve the model's generalization, it can also lead to larger models and increased computational resources for training and inference. Another area is image preprocessing, which involves performing image rotation correction (orthogonalizing) before inputting the image into the model. This can be achieved through image processing techniques such as affine transformation. However, image preprocessing also requires additional computing resources, or even dedicated circuits, which incurs additional computing resource or hardware costs. Therefore, existing convolutional neural network models or deep learning models still have shortcomings in image recognition applications. Summary of the Invention

[0003] In view of this, the present invention proposes a convolution operation circuit and related methods, which can improve the adaptability of deep learning models or convolutional neural network models to changes in the direction / rotation of images. In an embodiment of the present invention, the post-convolution kernel is selectively adjusted by judging the rotation state of the input image. When the input image has a non-zero rotation angle, a convolution operation is performed on the input image based on the adjusted convolution kernel. The advantage of the present invention is that it does not require additional computing resources or increase additional hardware costs to perform image preprocessing, nor does it require training the model for images with non-zero rotation angles, which results in an increase in model size.

[0004] An embodiment of the present invention provides a convolution operation circuit, comprising: a rotation determination unit, a convolution kernel adjustment unit, and a convolution operation unit. The rotation determination unit is configured to determine a rotation state of input data. The convolution kernel adjustment unit is coupled to the rotation determination unit and configured to selectively adjust an initial convolution kernel based on the rotation state to obtain an adjusted convolution kernel. The convolution operation unit is coupled to the convolution kernel adjustment unit and configured to perform a convolution operation on the input data based on the adjusted convolution kernel to obtain a feature map.

[0005] An embodiment of the present invention provides a convolution operation method, which includes: determining a rotation state of input data; selectively adjusting an initial convolution kernel according to the rotation state to obtain an adjusted convolution kernel; and performing a convolution operation on the adjusted convolution kernel and the input data to obtain a feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 FIG. 4 is a diagram showing the architecture of a convolution operation circuit according to an embodiment of the present invention.

[0007] Figure 2A and Figure 2B The rotation changes of the input data corresponding to the aligned image and the unaligned image are shown.

[0008] Figure 3A and Figure 3B It shows how an embodiment of the present invention uses an initial convolution kernel and an adjusted convolution kernel to perform convolution operations on input data of aligned images and unaligned images.

[0009] Figure 4A and Figure 4B It shows how an embodiment of the present invention uses the initial convolution kernel and the adjusted convolution kernel to perform convolution operations on padded input data of aligned images and unaligned images.

[0010] Figure 5 A flow chart of a convolution operation method according to an embodiment of the present invention is shown.

[0011] Explanation of symbols

[0012] 100 Convolution Operation Circuit

[0013] 110 Rotation judgment unit

[0014] 120 convolution kernel adjustment unit

[0015] 130 Convolution Operation Unit

[0016] 140 Filling Processing Unit

[0017] 150 mapping units DETAILED DESCRIPTION

[0018] In the following text, many specific details are described to provide the reader with a thorough understanding of the embodiments of the present invention. However, those skilled in the art will understand how to implement the present invention without one or more specific details, or using other methods, components, materials, etc. In other cases, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring the core concepts of the present invention.

[0019] References in this specification to "one embodiment" mean that the specific features, structures, or characteristics described in that embodiment may be included in at least one embodiment of the present invention. Therefore, the phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment. Furthermore, the aforementioned specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0020] The present invention proposes a convolution operation circuit and method for a convolution layer (Convolution Layer) or a convolution operation-related layer that can be used in a deep learning model or a convolutional neural network model. The circuit and method have good adaptability to input data, can effectively improve the efficiency of convolution operations on non-aligned images, and extract features from the input data more quickly for subsequent layers (such as pooling layers and fully connected layers) of the deep learning model or convolutional neural network model to perform classification, detection or other advanced tasks.

[0021] Please refer to Figure 1 , which shows a convolution operation circuit according to an embodiment of the present invention. As shown in the figure, the convolution operation circuit 100 includes: a rotation determination unit 110, a convolution kernel (or convolution filter) adjustment unit 120, a convolution operation unit 130, and a padding processing unit 140. The rotation determination unit 110 is used to determine a rotation state R_INFO of an input data IND. Please refer to Figure 2A and Figure 2B . Figure 2AThe captured image PIC_A can be one of an image or image sequence generated by an image capture device (e.g., a still camera or a motion picture camera) capturing a scene, and the input data IND can be pixel data corresponding to a block of the captured image PIC_A. Under different shooting conditions, the image capture device may be rotated, resulting in a captured image PIC_B that is rotated 90 degrees clockwise relative to the captured image PIC_A. Furthermore, in an embodiment of the present invention, the input data IND can be pixel data corresponding to a block of the captured image PIC_A, pixel data corresponding to a block of the captured image PIC_B that is rotated 90 degrees clockwise relative to the captured image PIC_A, or pixel data corresponding to a block of a non-aligned captured image that is rotated at a specific angle relative to the captured image PIC_A. The rotation status R_INFO generated by the rotation determination unit 110 indicates the rotation angle between the captured image to which the input data IND belongs and an aligned captured image (e.g., a captured image aligned on top of an image above the subject). Furthermore, in different embodiments of the present invention, the rotation determination unit 110 may determine and generate the rotation state R_INFO based on metadata of the captured image, device rotation information generated by the image capturing device, or features of specific elements in the captured image.

[0022] Furthermore, the convolution kernel adjustment unit 120 is used to adjust the initial convolution kernel CONK_INT according to the rotation state R_INFO, thereby obtaining the adjusted convolution kernel CONK_ADJ. Figure 3A and Figure 3B The convolution operation principle is demonstrated. Figure 3A In , the input data IND(1) is the pixel data of a block of the aligned acquired image (including pixel data 0 to 35). Therefore, the convolution operation on the input data IND(1) only needs to be performed through the initial convolution kernel CONK_INT (including convolution weights A, B, C, D, E, F, G, H, I). On the other hand, in Figure 3BIn the example, the input data IND(2) is pixel data of a block of non-aligned acquired images rotated 90 degrees clockwise (including pixel data 0 to 35 rotated 90 degrees clockwise). Therefore, the convolution operation on the input data IND(2) needs to be performed using the adjusted convolution kernel CONK_ADJ (including convolution weights A, B, C, D, E, F, G, H, and I rotated 90 degrees clockwise). Since the rotation state R_INFO of the input data IND(2) is 90 degrees clockwise, the convolution kernel adjustment unit 120 rotates the convolution weights A to I in the initial convolution kernel CONK_INT by 90 degrees clockwise to obtain the adjusted convolution kernel CONK_ADJ. It should be noted that as the rotation state R_INFO of the input data IND changes, the convolution kernel adjustment unit 120 will adaptively rotate the initial convolution kernel CONK_INT, rather than being limited to the aforementioned 90 degrees clockwise rotation. For example, if the rotation state R_INFO of the input data IND is 180 degrees clockwise, the convolution kernel adjustment unit 120 rotates the initial convolution kernel CONK_INT 180 degrees clockwise to obtain the adjusted convolution kernel CONK_ADJ.

[0023] The convolution operation unit 130 performs a convolution operation on the input data IND using the adjusted convolution kernel CONK_ADJ to obtain a convolution operation result. Please note that if the rotation state R_INFO of the input data IND indicates that the input data IND is aligned with the pixel data of the acquired image, the convolution operation unit 130 performs a convolution operation on the input data IND using the initial convolution kernel CONK_INT. Specifically, the convolution operation unit 130 performs the following operations: placing the initial convolution kernel CONK_INT or the adjusted convolution kernel CONK_ADJ at the first column (Column) and first row (Row) of the input data IND(1) or IND(2), calculating the dot product sum between the initial convolution kernel CONK_INT or the adjusted convolution kernel CONK_ADJ and the data it covers (i.e., the dot product sum between each pixel data and the corresponding convolution weight), generating an output value, and sliding the initial convolution kernel CONK_INT or the adjusted convolution kernel CONK_ADJ to the right by one stride unit along the row direction. For example, it moves right one unit in the row direction (the step unit is 1) or two units in the row direction (the step unit is 2). Then, the dot product sum between the initial convolution kernel CONK_INT or the adjusted convolution kernel CONK_ADJ and the data it covers is calculated again to generate another output value. When the initial convolution kernel CONK_INT or the adjusted convolution kernel CONK_ADJ moves to the end of the input data IND(1) or IND(2), it returns to the beginning in the column direction and slides down one step unit. For example, after returning to the first column and the first row in the column direction, it moves down one unit in the column direction (the step unit is 1) or two units in the column direction (the step unit is 2). This process is continued until the initial convolution kernel CONK_INT or the adjusted convolution kernel CONK_ADJ covers the entire input data IND(1) or IND(2).

[0024] When the convolution operation unit 130 performs a convolution operation using the adjusted convolution kernel CONK_ADJ and the input data IND, the generated convolution operation result may need to be processed by the mapping unit 150 in the convolution operation circuit 100 to obtain a convolution operation result (i.e., a feature map) that is consistent with the aligned acquired image. Please refer again to Figure 3A and Figure 3B In the example shown in the figure, the convolution operation is performed on the input data IND(2) using the adjusted convolution kernel CONK_ADJ rotated 90 degrees clockwise (e.g. Figure 3B ), the operation result corresponding to the operation step (0,0) is the same as the convolution operation performed on the input data IND(1) using the initial convolution kernel CONK_INT (as shown in FIG. Figure 3A), and the result of operation step (1,0) performed on the input data IND(2) using the adjusted convolution kernel CONK_ADJ is consistent with the result of operation step (0,2) performed on the input data IND(1) using the initial convolution kernel CONK_INT. Therefore, in one embodiment of the present invention, the mapping unit 150 can be used to process (map) the convolution operation result of the adjusted convolution kernel CONK_ADJ, thereby obtaining a feature map consistent with the convolution operation result using the initial convolution kernel CONK_INT.

[0025] However, it should be noted that the mapping unit 150 is not necessarily a necessary component. In some embodiments of the present invention, when the convolution operation unit 130 uses the adjusted convolution kernel CONK_ADJ to perform the convolution operation on the input data IND(2), an atypical operation order may be adopted, which does not follow the order from upper left to lower right. For example, the convolution operation unit 130 may directly start the first step from operation step (3,0) because the operation result of operation step (3,0) of the convolution operation on the input data IND(2) using the adjusted convolution kernel CONK_ADJ is exactly the same as the operation result of operation step (0,0) of the convolution operation on the input data IND(1) using the initial convolution kernel CONK_INT. Next, the convolution operation unit 130 may shift the adjusted convolution kernel CONK_ADJ downward by one unit and perform operation step (3, 1). The operation result of operation step (3, 1) is exactly the same as the operation result of operation step (1, 0) of performing the convolution operation on the input data IND(1) using the initial convolution kernel CONK_INT. Similarly, by using an atypical operation sequence, an operation result having the same operation sequence as that of performing the convolution operation on the input data IND(1) using the initial convolution kernel CONK_INT can be obtained. Therefore, by using an atypical operation sequence, the mapping unit 150 can also be ignored.

[0026] In some cases, when there are specific requirements for the size of the feature map generated by the convolution operation, the convolution operation circuit 100 may perform padding on the input data IND. For example, when it is necessary to obtain a feature map of the same size as the input data, padding is performed on the edges of the top row, bottom row, leftmost column, and rightmost column of the input data IND, and a convolution operation with a stride of 1 is performed. Alternatively, when it is necessary to obtain a feature map half the size of the input data, padding is performed on the edges of the bottom row and rightmost column of the input data IND, and a convolution operation with a stride of 2 is performed.

[0027] When the padding processing unit 140 in the convolution operation circuit 100 decides to perform padding processing on the input data IND, the padding processing unit 150 will consider the rotation state R_INFO of the input data IND. Figure 4A and Figure 4B Description. Figure 4A In the example shown in , when the padding processing unit 140 pads the input data IND(1) belonging to the aligned acquired image, the padding processing unit 150 adds padding data (using 0 for padding (Zero-Padding)) next to the rightmost column of the input data IND(1) and adds padding data below the bottom row of the input data IND(1), thereby obtaining the padded data IND(3). Figure 4B In the example shown in FIG, when the input data IND(2) belongs to an unaligned acquired image, the padding processing unit 150 will use a different Figure 4A The padding is performed according to the padding principle shown in (i.e., padding data is added to the right and below the input data). Further, since the rotation state R_INFO of the input data IND(2) has a rotation angle of 90 degrees clockwise relative to the input data IND(1) belonging to the aligned acquired image. Therefore, the padding processing unit 140 will not add padding data to the right and below the input data IND(2), but will add padding data (filled with 0) to the left of the leftmost column of the input data IND(2), and add padding data below the lowest row of the input data IND(2), thereby obtaining the padded data IND(4). Further, as the rotation state R_INFO of the input data IND is different, the padding processing unit 150 will add padding data at different positions. Compare Figure 4A and Figure 4B It can be seen that after data padding is performed based on this principle, the convolution operation result between the non-aligned padded input data IND(4) and the adjusted convolution kernel CONK_AD can be consistent with the convolution operation result between the aligned padded input data IND(3) and the initial convolution kernel CONK_INT.

[0028] Figure 5 The convolution operation method in an embodiment of the present invention is shown. As shown in the figure, the method includes the following simplified process:

[0029] S110: Determine a rotation state of input data;

[0030] S120: selectively adjusting an initial convolution kernel according to the rotation state, thereby obtaining an adjusted convolution kernel; and

[0031] S130: Perform a convolution operation on the input data based on the adjusted convolution kernel to obtain a feature map.

[0032] Since the principles and specific details of the above steps have been described in detail in the previous embodiments, they will not be repeated here. It is worth noting that the above process can be further improved by adding other additional steps or making appropriate modifications and adjustments to better achieve the computational efficiency of the convolution-related layers, thereby further improving the reasoning efficiency and performance of the deep learning model or convolutional neural network model.

[0033] It can be seen from the above embodiments that the convolution operation circuit and method of the present invention have good adaptability to the input data of non-aligned (rotated) images. That is to say, the convolution operation circuit and method of the present invention do not need to rely on additional image preprocessing processes to perform rotation correction on the input data of the non-aligned (rotated) image and to normalize the input data of the non-aligned (rotated) image, but can directly perform convolution operation on the input data of the non-aligned (rotated) image with the adjusted convolution kernel. In addition, the convolution operation circuit and method of the present invention do not need to expand the training data set to improve the adaptability of the model to image angle or rotation changes. That is to say, the convolution operation circuit and method of the present invention are applicable to any model trained with aligned (unrotated) images, and there is no need to retrain the model. Compared with the prior art, the convolution operation circuit and method of the present invention have lower requirements for computing resources and hardware, and are more suitable for many types of edge devices.

[0034] The above-mentioned embodiments of the present invention can be implemented in the form of devices, methods or computer program products. Therefore, the embodiments of the present invention can take different forms such as full hardware implementation, full software implementation or software and hardware combination implementation, and these different forms of implementation can be collectively referred to as "modules" or "systems" in this article. In addition, the embodiments of the present invention can be implemented in the form of a computer program product in any tangible medium, wherein the tangible medium contains computer-usable program code. In terms of hardware, the present invention can be implemented by any of the following technologies or related combinations, namely: individual operation logic (operation logic) having logic gates that can perform logical functions based on data signals, and application specific integrated circuits (ASICs), programmable gate arrays (PGAs) or field programmable gate arrays (FPGAs) with suitable combinational logic.

[0035] The flow charts and blocks in the specification illustrate the architecture, functions and operations that can be implemented by the system, method and computer software product based on various embodiments of the present invention. In this regard, each block in the flow chart or function block diagram can represent a module, section or part of a program code, which includes one or more executable instructions for implementing a specific logical function. In addition, each block in the function block diagram and / or flow chart, and the combination of blocks, can basically be implemented by a dedicated hardware system that performs a specified function or action, or a combination of dedicated hardware and computer program instructions. These computer program instructions can also be stored in a computer-readable medium, which can enable a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable medium are sufficient to implement the function / action specified by the blocks in the flow chart and / or function block diagram.

[0036] The above descriptions are merely preferred embodiments of the present invention. All equivalent changes and modifications made according to the claims of the present invention should fall within the scope of the present invention.

Claims

1. A convolution operation circuit, comprising: a rotation determination unit for determining a rotation state of input data; a convolution kernel adjustment unit, coupled to the rotation determination unit, for selectively adjusting an initial convolution kernel according to the rotation state, thereby obtaining an adjusted convolution kernel; and A convolution operation unit is coupled to the convolution kernel adjustment unit and is used to perform a convolution operation on the input data according to the adjusted convolution kernel, so as to obtain a feature map.

2. The convolution operation circuit as described in claim 1, wherein the rotation determination unit is used to determine the rotation state based on metadata of a captured image corresponding to the input data, device rotation information generated by an image capture device used to generate the captured image, and / or a feature of a specific element in the captured image.

3. The convolution operation circuit as claimed in claim 1, wherein the convolution kernel adjustment unit rotates the convolution weights in the initial convolution kernel by a rotation angle indicated by the rotation state to obtain the adjusted convolution kernel.

4. The convolution operation circuit according to claim 1, further comprising: a filling processing unit for performing filling processing on the input data according to the rotation state, thereby obtaining filled input data; in, The position of the padding data in the padded input data is related to the rotation state.

5. The convolution operation circuit as claimed in claim 4, wherein the convolution kernel adjustment unit is used to perform a convolution operation based on the adjusted convolution kernel and the padded input data to obtain the feature map.

6. The convolution operation circuit according to claim 2, further comprising: A mapping unit is used to perform a mapping process on a plurality of operation results generated by the convolution operation unit performing a convolution operation on the adjusted convolution kernel and the input data according to the device rotation information to generate the feature map.

7. A convolution operation method, comprising: Determining a rotation state of an input data; selectively adjusting an initial convolution kernel according to the rotation state to obtain an adjusted convolution kernel; and A convolution operation is performed on the input data based on the adjusted convolution kernel to obtain a feature map.

8. The convolution operation method as described in claim 7, wherein a rotation judgment unit is used to determine the rotation state based on metadata of a captured image corresponding to the input data, device rotation information generated by an image capture device used to generate the captured image, and / or a feature of a specific element in the captured image.

9. The convolution operation method of claim 7 , wherein the step of selectively adjusting the initial convolution kernel according to the rotation state comprises: According to a rotation angle indicated by the rotation state, the convolution weights in the initial convolution kernel are rotated by the rotation angle to obtain the adjusted convolution kernel.

10. The convolution operation method according to claim 7, further comprising: The input data is padded according to the rotation state, thereby obtaining padded input data, wherein the padded data position in the padded input data is related to the rotation state.

Images