Method, device, electronic equipment and storage medium for image processing

Abstract

The application belongs to the technical field of image processing, and discloses an image processing method, device, electronic equipment and storage medium. The method comprises the following steps: performing data padding on target feature map data according to a current convolution kernel column index, the target feature map data being selected from a target image and being stored in a data register; performing data offset processing on the padded target feature map data according to the convolution kernel column index, to obtain offset target feature map data; and performing operation based on convolution kernel data and the offset target feature map data, to obtain a feature map processing result. In this way, the operation is performed in the data offset manner, the number of data reading and writing is reduced, and the image processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to image processing methods, apparatus, electronic devices, and storage media. Background Technology

[0002] Image convolution operations have been widely applied in various fields. In some image processing scenarios, coprocessors are typically used to process target images based on depthwise separable convolutional networks, and the results are then used for image classification or object detection.

[0003] However, when processing target images using a coprocessor based on a depthwise separable convolutional network, the convolution operation is inefficient, resulting in low image processing efficiency.

[0004] Therefore, in application scenarios where image processing is performed using a coprocessor based on a depthwise separable convolutional network, there is a need for an image processing technology that can improve the efficiency of convolution operations and thus improve the efficiency of image processing. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for image processing, so as to improve the efficiency of convolution operations and thus improve the efficiency of image processing when image processing is performed by a coprocessor based on a depthwise separable convolutional network.

[0006] On the one hand, an image processing method is provided, including:

[0007] Based on the current convolution kernel column index, the target feature map data is padded with data. The target feature map data is the data to be processed selected from the target image and stored in the data register.

[0008] Based on the column index of the convolution kernel, the padded target feature map data is shifted to obtain the shifted target feature map data.

[0009] The feature map processing result is obtained by performing calculations based on the convolution kernel data and the offset target feature map data.

[0010] In one implementation, data padding is performed on the target feature map data based on the current convolution kernel column index, including:

[0011] If the kernel column index is determined to be the start or end value of the convolution column, then obtain the current position of the image pointer pointed to by the target image pointer;

[0012] Data is filled into the target feature map data based on the image pointer position.

[0013] In one implementation, the image pointer position includes an image column index, and data filling of the target feature map data is performed according to the image pointer position, including:

[0014] If the kernel column index is determined to be the starting value of the convolution column, and the image column index is determined to be the starting value of the image column, then m zeros are filled into the front-end register to obtain the padded target feature map data; m is a natural number; the front-end register is the register preceding the data register;

[0015] If the kernel column index is determined to be the starting value of the convolution column, and the image column index is not the starting value of the image column, then the m values ​​before the image pointer position are filled into the front-end register to obtain the filled target feature map data.

[0016] If the kernel column index is determined to be the convolution column termination value, and the image column index meets the column termination condition, then k zeros are added to the last k bits of the data register to obtain the padded target feature map data; k is a natural number.

[0017] If the convolution kernel column index is determined to be the convolution column termination value, and the image column index does not meet the column termination condition, then based on the image pointer position and the data length of the target feature map data, select data of length k from the target image to be filled, and fill the data to be filled into the last k bits of the data register to obtain the filled target feature map data.

[0018] In one implementation, the padded target feature map data is shifted according to the column index of the convolution kernel to obtain the shifted target feature map data, including:

[0019] If the kernel column index is determined to be the starting value of the convolution column, then the padded target feature map data is shifted left by m-1 positions;

[0020] If it is determined that the kernel column index is not the starting value of the convolution column, then the padded target feature map data is shifted left by 1 bit.

[0021] In one implementation, after performing calculations based on the convolution kernel data and the offset target feature map data to obtain the feature map processing result, the method further includes:

[0022] Update the kernel index information until the conditions for completing the operation are met. The kernel index information includes the kernel column index.

[0023] Based on the current convolution kernel column index, the target feature map data is padded, including:

[0024] If it is determined that the convolution kernel index information has been updated, then the target feature map data is populated based on the updated convolution kernel index information.

[0025] In one implementation, the image pointer position includes the image row index, and updating the convolution kernel index information includes:

[0026] If it is determined that the kernel column index is lower than the convolution column termination value, then increment the kernel column index by one;

[0027] If the kernel column index is determined to be the convolution column termination value and the kernel row index is the total number of kernel rows, then the conditions for the completion of the convolution kernel operation on the target feature map data are met.

[0028] If the kernel column index is determined to be the end value of the convolution column, and the kernel row index is lower than the total number of convolution rows, then update the kernel column index to the start value of the convolution column, increment the kernel row index by one, and increment the image row index by one.

[0029] On one hand, an image processing apparatus is provided, comprising:

[0030] The padding unit is used to pad the target feature map data according to the current convolution kernel column index. The target feature map data is the data to be processed selected from the target image and stored in the data register.

[0031] The offset unit is used to perform data offset processing on the padded target feature map data according to the column index of the convolution kernel to obtain the offset target feature map data;

[0032] The computation unit is used to perform operations based on the convolution kernel data and the offset target feature map data to obtain the feature map processing results.

[0033] In one embodiment, the filling unit is used for:

[0034] If the kernel column index is determined to be the start or end value of the convolution column, then obtain the current position of the image pointer pointed to by the target image pointer;

[0035] Data is filled into the target feature map data based on the image pointer position.

[0036] In one implementation, the image pointer position includes an image column index, and the padding unit is used for:

[0037] If the kernel column index is determined to be the starting value of the convolution column, and the image column index is determined to be the starting value of the image column, then m zeros are filled into the front-end register to obtain the padded target feature map data; m is a natural number; the front-end register is the register preceding the data register;

[0038] If the kernel column index is determined to be the starting value of the convolution column, and the image column index is not the starting value of the image column, then the m values ​​before the image pointer position are filled into the front-end register to obtain the filled target feature map data.

[0039] If the kernel column index is determined to be the convolution column termination value, and the image column index meets the column termination condition, then k zeros are added to the last k bits of the data register to obtain the padded target feature map data; k is a natural number.

[0040] If the convolution kernel column index is determined to be the convolution column termination value, and the image column index does not meet the column termination condition, then based on the image pointer position and the data length of the target feature map data, select data of length k from the target image to be filled, and fill the data to be filled into the last k bits of the data register to obtain the filled target feature map data.

[0041] In one implementation, the offset unit is used for:

[0042] If the kernel column index is determined to be the starting value of the convolution column, then the padded target feature map data is shifted left by m-1 positions;

[0043] If it is determined that the kernel column index is not the starting value of the convolution column, then the padded target feature map data is shifted left by 1 bit.

[0044] In one embodiment, the arithmetic unit is further configured to:

[0045] Update the kernel index information until the conditions for completing the operation are met. The kernel index information includes the kernel column index.

[0046] Filling cells are used for:

[0047] If it is determined that the convolution kernel index information has been updated, then the target feature map data is populated based on the updated convolution kernel index information.

[0048] In one embodiment, the image pointer position includes an image row index, and the arithmetic unit is further used for:

[0049] If it is determined that the kernel column index is lower than the convolution column termination value, then increment the kernel column index by one;

[0050] If the kernel column index is determined to be the convolution column termination value and the kernel row index is the total number of kernel rows, then the conditions for the completion of the convolution kernel operation on the target feature map data are met.

[0051] If the kernel column index is determined to be the end value of the convolution column, and the kernel row index is lower than the total number of convolution rows, then update the kernel column index to the start value of the convolution column, increment the kernel row index by one, and increment the image row index by one.

[0052] On one hand, an electronic device is provided, including a processor and a memory storing computer-readable instructions that, when executed by the processor, perform the steps of the method provided in various alternative implementations of any of the above-described image processing methods.

[0053] On the one hand, a computer-readable storage medium is provided on which a computer program is stored, which, when executed by a processor, performs the steps of the method provided in various alternative implementations of any of the above-described image processing methods.

[0054] On the one hand, a computer program product is provided that, when run on a computer, causes the computer to perform the steps of the method provided in any of the various alternative implementations of the image processing described above.

[0055] The image processing method, apparatus, electronic device, and storage medium provided in this application embodiment include: filling target feature map data with data according to the current convolution kernel column index; the target feature map data being the data to be processed selected from the target image and stored in a data register; performing data offset processing on the filled target feature map data according to the convolution kernel column index to obtain offset target feature map data; and performing calculations based on the convolution kernel data and the offset target feature map data to obtain the feature map processing result. This data offset method reduces the number of data read / write operations and improves image processing efficiency.

[0056] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this application;

[0059] Figure 2 A flowchart illustrating a single-channel image processing method provided in this application embodiment;

[0060] Figure 3A flowchart illustrating an example of convolution kernel operation provided in this application embodiment;

[0061] Figure 4 A structural block diagram of an image processing apparatus provided in an embodiment of this application;

[0062] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0064] First, some of the terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0065] Terminal devices can be mobile terminals, fixed terminals, or portable terminals, such as mobile phones, sites, units, devices, multimedia computers, multimedia tablets, internet nodes, communicators, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, personal communication system devices, personal navigation devices, personal digital assistants, audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. It is also foreseeable that terminal devices can support any type of user-facing interface (e.g., wearable devices).

[0066] Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0067] In order to improve the efficiency of convolution operations and thus the efficiency of image processing when performing image processing based on depthwise separable convolutional networks using a coprocessor, embodiments of this application provide a method, apparatus, electronic device, and storage medium for image processing.

[0068] In this embodiment of the application, the coprocessor is applied in an electronic device, which can be a server or a terminal device.

[0069] See Figure 1 The diagram shown is a flowchart of an image processing method provided in an embodiment of this application. The specific implementation flow of this method is as follows:

[0070] Step 100: Fill the target feature map data with data based on the current convolution kernel column index.

[0071] The target feature map data consists of data selected from the target image and stored in a data register.

[0072] In one embodiment, the target image is an image captured by a camera device.

[0073] In one embodiment, the target image is an image generated after converting speech data or sequence data into an image.

[0074] Step 101: Based on the column index of the convolution kernel, perform data offset processing on the padded target feature map data to obtain the offset target feature map data.

[0075] Step 102: Perform calculations based on the convolution kernel data and the offset target feature map data to obtain the feature map processing results.

[0076] The embodiments of this application can be applied to scenarios of image classification or target tracking. One application scenario is: through steps 100-102, the image is processed to obtain the image processing result, and the image is classified or the target is tracked based on the image processing result.

[0077] Furthermore, after determining the feature map processing result, the convolution kernel index information is updated until the operation completion condition is met. The convolution kernel index information includes the convolution kernel column index. If it is determined that the convolution kernel index information needs to be updated, then the updated convolution kernel index information is used to update the kernel index information, and step 100 is executed.

[0078] To achieve data filling on the left and right sides of the target image, the implementation process of step 100 includes: if the kernel column index is determined to be the start value or end value of the convolution column, then the image pointer position currently pointed to by the target image pointer is obtained, and data filling is performed on the target feature map data according to the image pointer position.

[0079] In one implementation, step 100 can be performed in any of the following ways:

[0080] Method 1: If the kernel column index is determined to be the starting value of the convolution column, then obtain the current image pointer position pointed to by the target image pointer, and fill the target feature map data according to the image pointer position.

[0081] Method 2: If the kernel column index is determined to be the convolution column termination value, then obtain the current image pointer position pointed to by the target image pointer, and fill the target feature map data according to the image pointer position.

[0082] Method 3: If the convolution kernel column index is determined to be an intermediate index value, then no data padding is performed on the target feature map data.

[0083] Among them, the intermediate index value (e.g., 5) is higher than the starting value of the convolution column and lower than the ending value of the convolution column.

[0084] To obtain the target feature map data to be processed, in one embodiment, if the kernel column index is determined to be the starting value of the convolution column, then based on the image pointer position, n consecutive numbers (e.g., 32) are selected from the target image, where n is a natural number; and the target feature map data is stored in the data register.

[0085] The image pointer position is the current position in the target image pointed to by the target image pointer, and can include the image row index and the image column index.

[0086] For example, if the kernel column index is 0, then if the kernel column index is determined to be the starting value of the convolution column, then the image pointer position currently pointed to by the target image pointer is obtained, i.e. (0, 0). Then, the values ​​of column indices 0-31 (i.e. target feature map data) are selected from the 0th row of the target image, and the selected target feature map data is added to the data register.

[0087] In one implementation method, when executing method 1, the following steps may be adopted:

[0088] If the kernel column index is determined to be the starting value of the convolution column, and the image column index is determined to be the starting value of the image column, then m zeros are filled into the front-end register to obtain the padded target feature map data; m is a natural number; the front-end register is the register preceding the data register;

[0089] If the kernel column index is determined to be the starting value of the convolution column, and the image column index is not the starting value of the image column, then the m values ​​before the image pointer position are filled into the front-end register to obtain the filled target feature map data.

[0090] For example, the data registers are BECO_REG1 to BECO_REG4. The front-side register is BECO_REG0, and BECO_REG is a general-purpose register in the coprocessor (Beco).

[0091] In one implementation method, when executing method 2, the following steps can be taken:

[0092] If the kernel column index is determined to be the convolution column termination value, and the image column index meets the column termination condition, then k zeros are added to the last k bits of the data register to obtain the padded target feature map data; k is a natural number.

[0093] If the convolution kernel column index is determined to be the convolution column termination value, and the image column index does not meet the column termination condition, then based on the image pointer position and the data length of the target feature map data, select data of length k from the target image to be filled, and fill the data to be filled into the last k bits of the data register to obtain the filled target feature map data.

[0094] In one implementation, if it is determined that the image column index + n > the image row termination value, then the image pointer position is determined to meet the column termination condition.

[0095] Among them, the convolution column termination value is the maximum value of the convolution kernel column index, and the image row termination value is the maximum value of the target image column index.

[0096] For example, if k is 8, the data registers are BECO_REG1 to BECO_REG4, which is a total of four registers. Each register contains 8 bits, so data can be filled into BECO_REG4.

[0097] To offset the target feature map data for subsequent convolution operations, in one implementation, step 101 can be performed in any of the following ways:

[0098] If the kernel column index is determined to be the starting value of the convolution column, the padded target feature map data is shifted left by m-1 positions; if the kernel column index is determined not to be the starting value of the convolution column, the padded target feature map data is shifted left by 1 position.

[0099] For example, if m is 8, then shift left by 7 bits.

[0100] To perform loop operations on the target image, step 102 may include:

[0101] If it is determined that the kernel column index is lower than the convolution column termination value, then increment the kernel column index by one and proceed to step 100.

[0102] If the kernel column index is determined to be the convolution column termination value and the kernel row index is the total number of kernel rows, then the conditions for the completion of the convolution kernel operation on the target feature map data are met.

[0103] If the kernel column index is determined to be the convolution column termination value, and the kernel row index is lower than the total number of kernel rows, then update the kernel column index to the convolution column start value, increment the kernel row index by one, and increment the image row index by one, and execute step 100.

[0104] Optionally, the initial values ​​for the kernel row index and the kernel column index can be 0.

[0105] As an example, assume the convolution column termination value is 3, the total number of convolution kernel rows is 3, the image row start value is 0, the current convolution kernel row index is 0, and the current image row index is 0. If the convolution kernel column index is 2, then the convolution kernel column index 2 + 1 = 3, and step 100 is executed. If the convolution kernel column index is determined to be 3, then the convolution kernel column index is set to 0, the image row index is updated to 0 + 1 = 1, and the convolution kernel row index is updated to 0 + 1 = 1.

[0106] In this way, after processing one row of data in the target feature map data, the row index can be updated so that the next row of data can be processed.

[0107] In practical applications, the conditions for completing the calculation can be set according to the actual application scenario.

[0108] In one implementation, the process of determining whether the convolution kernel index information meets the conditions for completion of the operation may include:

[0109] When the row index of the convolution kernel is determined to be the total number of rows in the convolution kernel, it is determined that the conditions for completing the operation are met.

[0110] In this way, a single operation of the convolution kernel can be performed on the target feature map data.

[0111] Furthermore, after determining that the conditions for completion of the operation are met, the image index information can be updated, and based on the updated image index information and the updated convolution kernel index information, steps 100-102 are executed cyclically to achieve convolution operation on the entire target image.

[0112] In one implementation, the image column index is incremented by n. If the updated image column index is not greater than the image column termination value, the image row index is set to the image row start value, the convolution kernel row index is updated to the convolution row start value, and the convolution kernel column index is set to the convolution column start value. Steps 100-102 are then executed. Here, n is a natural number.

[0113] If the updated image column index is greater than the image column termination value and the image row start value is lower than the image row output threshold, then increment the image row start value by one, set the image row index to the updated image row start value, set the image column index to the image column start value, set the convolution kernel row index to the convolution row start value, and set the convolution kernel column index to the convolution column start value, and execute steps 100-102.

[0114] If the updated image column index is greater than the image column termination value, and the image row start value is the image row output threshold, then the single-channel image operation is considered complete.

[0115] Furthermore, if it is determined that the single-channel image operation is complete, convolution operations can be performed on other channels of the target image.

[0116] In practical applications, the image row output threshold can be set according to the actual application scenario. For example, the image row output threshold = (total number of rows in the target image - total number of rows in the convolution kernel + number of padding rows) / convolution stride + 1.

[0117] In one implementation, if it is determined that the single-channel operation of the image is completed, it is determined whether the channel index is less than the channel index threshold. If so, the channel index is incremented by one, the image index information and the convolution kernel index information are initialized, and steps 100-102 are executed; otherwise, it is determined that the convolution operation of the target image is completed.

[0118] The initialization of image index information and convolution kernel index information includes: setting the image row start value to an initial value (e.g., 0), setting the image row index to the updated image row start value, setting the image column index to the image column start value, setting the convolution kernel row index to the convolution row start value, and setting the convolution kernel column index to the convolution column start value.

[0119] The channel index threshold is the maximum value of the channel index of the target image, such as 3.

[0120] See Figure 2 The diagram shown is a flowchart of a method for single-channel image convolution. Combined with... Figure 1 and 2 The single-channel convolution operation for images is explained below, and the specific implementation process of this method is as follows:

[0121] Step 200: Fill the target feature map data stored in the data register with data according to the current convolution kernel column index.

[0122] Step 201: Based on the column index of the convolution kernel, perform data offset processing on the padded target feature map data to obtain the offset target feature map data.

[0123] Step 202: Perform calculations based on the convolution kernel data and the offset target feature map data to obtain the feature map processing results.

[0124] Step 203: Determine whether the kernel column index is lower than the convolution column termination value. If yes, proceed to step 204; otherwise, proceed to step 205.

[0125] Step 204: Increment the kernel column index by one, and then proceed to step 200.

[0126] Step 205: Determine if the convolution kernel row index is lower than the total number of rows in the convolution kernel. If yes, proceed to step 206; otherwise, proceed to step 207.

[0127] Step 206: Update the kernel column index to the starting value of the convolution column, increment the kernel row index by one, and increment the image row index by one. Then execute step 200.

[0128] Step 207: Increment n by the index of the image column indicated by the target image pointer.

[0129] Step 208: Determine whether the updated image column index is not greater than the image column termination value. If yes, proceed to step 209; otherwise, proceed to step 210.

[0130] Step 209: Set the image row index to the image row start value, update the convolution kernel row index to the convolution row start value, set the convolution kernel column index to the convolution column start value, and execute step 200.

[0131] Step 210: Determine whether the starting value of the image row is lower than the image row output threshold. If yes, proceed to step 211; otherwise, proceed to step 212.

[0132] Step 211: Increment the image row starting value by one, set the image row index to the updated image row starting value, set the image column index to the image column starting value, set the kernel row index to the convolution row starting value, and set the kernel column index to the convolution column starting value. Then execute step 200.

[0133] Step 212: Confirm that the single-channel image processing is complete.

[0134] Furthermore, if it is determined that the single-channel image operation is complete, it can also be determined whether the channel index is less than the channel index threshold. If so, the channel index is incremented by one, and the image index information and convolution kernel index information are initialized; otherwise, it is determined that the target image convolution operation is complete.

[0135] In this embodiment, each starting value and each ending value can be set according to the actual application scenario, and no restrictions are imposed here.

[0136] As an example, kernel = 3x3, stride = 1, padding = 1, and data type q7 indicates 8 bits. kernel refers to the convolution kernel, stride is the stride, and padding is the number of rows padded per side. The convolution column starts at 0 and ends at 2. The length n of the target feature data selected in a single pass is 32, and the lengths m and k of the padding data are both 8. See also... Figure 3 The diagram shown is a flowchart of an example of convolution kernel operation. Then, combined with... Figure 3 right Figure 1 The convolution operation method in the example is illustrated.

[0137] Step 301: Determine if the kernel column index is 0. If yes, proceed to step 302; otherwise, proceed to step 306.

[0138] The kernel column index can be represented as i_ker_w, for example, i_ker_w = 0.

[0139] Step 302: Based on the image pointer position, select 32 consecutive values ​​from the target image to obtain the target feature map data, and store the target feature map data in the data register.

[0140] The image pointer position includes the image row index and the image column index.

[0141] Step 303: Determine if the image pointer position is the start position of the row. If yes, proceed to step 304; otherwise, proceed to step 305.

[0142] For example, if the image row index is 0, then it is determined as the starting position of the row.

[0143] Step 304: Fill the front-end register with 8 zeros to obtain the filled target feature map data, and then shift the filled target feature map data to the left by 7 bits.

[0144] For example, store target feature map data in data registers BECO_REG1 to BECO_REG4; fill the front-end register BECO_REG0 with 8 zeros to obtain the filled target feature map data stored in BECO_REG0 to BECO_REG4.

[0145] This allows for data filling at the beginning of image rows.

[0146] Step 305: Fill the front-end register with the 8 values ​​before the image pointer position to obtain the filled target feature map data, and then shift the filled target feature map data to the left by 7 bits.

[0147] Step 306: Determine if the kernel column index is 1. If yes, proceed to step 307; otherwise, proceed to step 308.

[0148] Step 307: Shift the target feature map data one bit to the left.

[0149] Step 308: Determine whether the image pointer position meets the column termination condition. If yes, proceed to step 309; otherwise, proceed to step 310.

[0150] Step 309: Fill the last 8 bits of the data register with 8 zeros to obtain the filled target feature map data, and then shift the filled target feature map data to the left by 1 bit.

[0151] Step 310: Select the 8 values ​​after the image column index + 32 and fill them into the last 8 bits of the data register to obtain the filled target feature map data, and then shift the filled target feature map data to the left by 1 bit.

[0152] This allows for data padding at the end of image rows.

[0153] In this embodiment, the coprocessor can be an embedded neural network processing unit (NPU). It can also be a Beco, whose core is a matrix multiplication unit, which includes 64 logic operation units capable of parallel computation of up to 64 multiplication-accumulation operations. Convolution operations can be performed using this matrix multiplication unit. Beco is configured with multiple registers, which, along with the matrix multiplication unit, enable data reading, writing, and computation. Furthermore, by configuring the data type, the target image and the convolution kernel can be made to have the same data type, thus supporting mixed-precision computation.

[0154] Table 1 shows a comparison table of image processing time.

[0155]

[0156] In traditional methods, image processing is usually based on the Cortex Microcontroller Software Interface Standard (CMSIS). Due to the use of CMSIS, a maximum of four outputs can be calculated in parallel in the channel direction. However, in the embodiments of this application, taking the Beco coprocessor as an example, parallel calculations with a larger coefficient can be performed in the feature map width direction, which improves the image processing performance by several times and greatly enhances the acceleration efficiency.

[0157] In this embodiment, multiple values ​​are selected from the image width direction each time, and they are multiplied and accumulated with each value in the convolution kernel in sequence (e.g., 32 values ​​can be calculated in parallel by Beco), which accelerates the data processing. Furthermore, after storing the target feature data in the data register, each operation only needs to shift the data in the register, instead of reading and writing a large amount of data each time, which greatly reduces the time cost and improves the efficiency of image processing.

[0158] Based on the same inventive concept, this application also provides an image processing apparatus. Since the principle of the above apparatus and device in solving the problem is similar to that of an image processing method, the implementation of the above apparatus can refer to the implementation of the method, and the repeated parts will not be described again.

[0159] like Figure 4 As shown, it is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application, including:

[0160] The padding unit 401 is used to pad the target feature map data according to the current convolution kernel column index. The target feature map data is the data to be processed selected from the target image and stored in the data register.

[0161] Offset unit 402 is used to perform data offset processing on the padded target feature map data according to the column index of the convolution kernel to obtain the offset target feature map data;

[0162] The operation unit 403 is used to perform operations based on the convolution kernel data and the offset target feature map data to obtain the feature map processing result.

[0163] In one embodiment, the filling unit 401 is used for:

[0164] If the kernel column index is determined to be the start or end value of the convolution column, then obtain the current position of the image pointer pointed to by the target image pointer;

[0165] Data is filled into the target feature map data based on the image pointer position.

[0166] In one embodiment, the image pointer position includes an image column index, and the padding unit 401 is used for:

[0167] If the kernel column index is determined to be the starting value of the convolution column, and the image column index is determined to be the starting value of the image column, then m zeros are filled into the front-end register to obtain the padded target feature map data; m is a natural number; the front-end register is the register preceding the data register;

[0168] If the kernel column index is determined to be the starting value of the convolution column, and the image column index is not the starting value of the image column, then the m values ​​before the image pointer position are filled into the front-end register to obtain the filled target feature map data.

[0169] If the kernel column index is determined to be the convolution column termination value, and the image column index meets the column termination condition, then k zeros are added to the last k bits of the data register to obtain the padded target feature map data; k is a natural number.

[0170] If the convolution kernel column index is determined to be the convolution column termination value, and the image column index does not meet the column termination condition, then based on the image pointer position and the data length of the target feature map data, select data of length k from the target image to be filled, and fill the data to be filled into the last k bits of the data register to obtain the filled target feature map data.

[0171] In one embodiment, the offset unit 402 is used for:

[0172] If the kernel column index is determined to be the starting value of the convolution column, then the padded target feature map data is shifted left by m-1 positions;

[0173] If it is determined that the kernel column index is not the starting value of the convolution column, then the padded target feature map data is shifted left by 1 bit.

[0174] In one embodiment, the arithmetic unit 403 is further configured to:

[0175] Update the kernel index information until the conditions for completing the operation are met. The kernel index information includes the kernel column index.

[0176] Filling unit 401 is used for:

[0177] If it is determined that the convolution kernel index information has been updated, then the target feature map data is populated based on the updated convolution kernel index information.

[0178] In one embodiment, the image pointer position includes the image row index, and the arithmetic unit 403 is further configured to:

[0179] If it is determined that the kernel column index is lower than the convolution column termination value, then increment the kernel column index by one;

[0180] If the kernel column index is determined to be the convolution column termination value and the kernel row index is the total number of kernel rows, then the conditions for the completion of the convolution kernel operation on the target feature map data are met.

[0181] If the kernel column index is determined to be the end value of the convolution column, and the kernel row index is lower than the total number of convolution rows, then update the kernel column index to the start value of the convolution column, increment the kernel row index by one, and increment the image row index by one.

[0182] The image processing method, apparatus, electronic device, and storage medium provided in this application embodiment include: filling target feature map data with data according to the current convolution kernel column index; the target feature map data being the data to be processed selected from the target image and stored in a data register; performing data offset processing on the filled target feature map data according to the convolution kernel column index to obtain offset target feature map data; and performing calculations based on the convolution kernel data and the offset target feature map data to obtain the feature map processing result. This data offset method reduces the number of data read / write operations and improves image processing efficiency.

[0183] Figure 5 A schematic diagram of the structure of an electronic device 5000 is shown. (See also...) Figure 5 As shown, the electronic device 5000 includes a processor 5010 and a memory 5020, and optionally may also include a power supply 5030, a display unit 5040, and an input unit 5050.

[0184] The processor 5010 is the control center of the electronic device 5000. It connects various components through various interfaces and lines, and performs various functions of the electronic device 5000 by running or executing software programs and / or data stored in the memory 5020, thereby performing overall monitoring of the electronic device 5000.

[0185] In this embodiment, when the processor 5010 calls the computer program stored in the memory 5020, it executes the steps in the above embodiments.

[0186] Optionally, processor 5010 may include one or more processing units; preferably, processor 5010 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 5010. In some embodiments, the processor and memory may be implemented on a single chip; in some embodiments, they may also be implemented separately on independent chips.

[0187] The memory 5020 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, various applications, etc.; the data storage area may store data created based on the use of the electronic device 5000, etc. In addition, the memory 5020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device, etc.

[0188] Electronic device 5000 also includes a power supply 5030 (such as a battery) that supplies power to various components. The power supply can be logically connected to processor 5010 through a power management system, thereby enabling the management of charging, discharging, and power consumption.

[0189] The display unit 5040 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 5000. In this embodiment of the invention, it is mainly used to display the display interfaces of various applications in the electronic device 5000, as well as text, images, and other objects displayed on the display interfaces. The display unit 5040 may include a display panel 5041. The display panel 5041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0190] The input unit 5050 can be used to receive information such as numbers or characters input by the user. The input unit 5050 may include a touch panel 5051 and other input devices 5052. The touch panel 5051, also known as a touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 5051).

[0191] Specifically, the touch panel 5051 can detect user touch operations and the signals generated by these operations, convert them into touch point coordinates, send them to the processor 5010, and receive and execute commands from the processor 5010. Furthermore, the touch panel 5051 can be implemented using various types of sensors, including resistive, capacitive, infrared, and surface acoustic wave sensors. Other input devices 5052 can include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0192] Of course, the touch panel 5051 can cover the display panel 5041. When the touch panel 5051 detects a touch operation on or near it, it transmits the information to the processor 5010 to determine the type of touch event. Subsequently, the processor 5010 provides corresponding visual output on the display panel 5041 according to the type of touch event. Although in Figure 5 In this embodiment, the touch panel 5051 and the display panel 5041 are two separate components to realize the input and output functions of the electronic device 5000. However, in some embodiments, the touch panel 5051 and the display panel 5041 can be integrated to realize the input and output functions of the electronic device 5000.

[0193] The electronic device 5000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity sensor, etc. Of course, depending on the specific application, the electronic device 5000 may also include other components such as a camera. Since these components are not the focus of this application's embodiments, therefore... Figure 5 It is not shown in the text and will not be described in detail here.

[0194] Those skilled in the art will understand that Figure 5 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or a combination of certain components, or different components.

[0195] In this embodiment of the application, a computer-readable storage medium stores a computer program thereon. When the computer program is executed by a processor, it enables a communication device to perform the various steps in the above embodiments.

[0196] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0197] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0199] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0200] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0201] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0202] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method, characterized in that, include: Based on the current convolution kernel column index, the target feature map data is filled with data, which is the data to be processed selected from the target image and stored in the data register; Based on the convolution kernel column index, the padded target feature map data is subjected to data offset processing to obtain the offset target feature map data; The feature map processing result is obtained by performing calculations based on the convolution kernel data and the offset target feature map data. The step of filling the target feature map data with data according to the current convolution kernel column index includes: if the convolution kernel column index is determined to be the start value or end value of the convolution column, then obtaining the image pointer position currently pointed to by the target image pointer; and filling the target feature map data with data according to the image pointer position. Furthermore, the image pointer position includes an image column index, and the step of filling the target feature map data according to the image pointer position includes: If the kernel column index is determined to be the starting value of the convolution column, and the image column index is determined to be the starting value of the image column, then m zeros are added to the front-end register to obtain the filled target feature map data; m is a natural number; the front-end register is the register preceding the data register. If the kernel column index is determined to be the starting value of the convolution column, and the image column index is not determined to be the starting value of the image column, then m values ​​before the image pointer position are added to the front-end register to obtain the filled target feature map data. If the kernel column index is determined to be the ending value of the convolution column, and the image column index meets the column termination condition, then k zeros are added to the last k positions of the data register to obtain the filled target feature map data; k is a natural number. If the convolution kernel column index is determined to be the convolution column termination value, and the image column index does not meet the column termination condition, then according to the image pointer position and the data length of the target feature map data, a data of length k to be filled is selected from the target image, and the data to be filled is filled into the last k bits of the data register to obtain the filled target feature map data. If the convolution kernel column index is determined to be an intermediate index value, then no data padding is performed on the target feature map data; Furthermore, the step of performing data offset processing on the padded target feature map data according to the convolution kernel column index to obtain the offset target feature map data includes: if it is determined that the convolution kernel column index is the starting value of the convolution column, then the padded target feature map data is shifted left by m-1 positions; if it is determined that the convolution kernel column index is not the starting value of the convolution column, then the padded target feature map data is shifted left by 1 position.

2. The method as described in claim 1, characterized in that, After performing operations based on the convolution kernel data and the offset target feature map data to obtain the feature map processing result, the method further includes: Update the convolution kernel index information until the operation completion condition is met. The convolution kernel index information includes the convolution kernel column index. The step of filling the target feature map data according to the current convolution kernel column index includes: If it is determined that the convolution kernel index information has been updated, then the target feature map data is populated according to the updated convolution kernel index information.

3. The method as described in claim 2, characterized in that, The image pointer position includes the image row index, and updating the convolution kernel index information includes: If it is determined that the kernel column index is lower than the convolution column termination value, then the kernel column index is incremented by one; If the convolution kernel column index is determined to be the convolution column termination value and the convolution kernel row index is the total number of convolution kernel rows, then the operation completion condition for the convolution kernel operation on the target feature map data is determined to be met. If it is determined that the convolution kernel column index is the convolution column termination value, and the convolution kernel row index is lower than the total number of convolution kernel rows, then the convolution kernel column index is updated to the convolution column start value, the convolution kernel row index is incremented by one, and the image row index is incremented by one.

4. An image processing apparatus, characterized in that, include: A padding unit is used to pad the target feature map data according to the current convolution kernel column index. The target feature map data is the data to be processed selected from the target image and stored in the data register. The offset unit is used to perform data offset processing on the padded target feature map data according to the column index of the convolution kernel to obtain the offset target feature map data. The computation unit is used to perform operations based on the convolution kernel data and the offset target feature map data to obtain the feature map processing results. The filling unit is used to: if the kernel column index is determined to be the start value or end value of the convolution column, obtain the current image pointer position pointed to by the target image pointer; and fill the target feature map data according to the image pointer position. The image pointer position includes the image column index, and the padding unit is used to: if the convolution kernel column index is determined to be the starting value of the convolution column, and the image column index is the starting value of the image column, then pad the front-end register with m zeros to obtain the padded target feature map data; m is a natural number; The front-end register is the register preceding the data register; If the convolution kernel column index is determined to be the starting value of the convolution column, and the image column index is not the starting value of the image column, then the m values ​​before the image pointer position are filled into the front-end register to obtain the filled target feature map data; if the convolution kernel column index is determined to be the ending value of the convolution column, and the image column index meets the column termination condition, then k zeros are filled into the last k positions of the data register to obtain the filled target feature map data; k is a natural number; if the convolution kernel column index is determined to be the ending value of the convolution column, and the image column index does not meet the column termination condition, then according to the image pointer position and the data length of the target feature map data, data of length k to be filled is selected from the target image, and the data to be filled is filled into the last k positions of the data register to obtain the filled target feature map data. The filling unit is further configured to: if the convolution kernel column index is determined to be an intermediate index value, then not to fill the target feature map data; The offset unit is used to: if the kernel column index is determined to be the starting value of the convolution column, shift the padded target feature map data to the left by m-1 positions; if the kernel column index is determined not to be the starting value of the convolution column, shift the padded target feature map data to the left by 1 position.

5. The apparatus as described in claim 4, characterized in that, The arithmetic unit is also used for: Update the convolution kernel index information until the operation completion condition is met. The convolution kernel index information includes the convolution kernel column index. The filling unit is used for: If it is determined that the convolution kernel index information has been updated, then the target feature map data is populated according to the updated convolution kernel index information.

6. The apparatus as claimed in claim 5, characterized in that, The image pointer position includes the image row index, and the arithmetic unit is further used for: If it is determined that the kernel column index is lower than the convolution column termination value, then the kernel column index is incremented by one; If the convolution kernel column index is determined to be the convolution column termination value and the convolution kernel row index is the total number of convolution kernel rows, then the operation completion condition for the convolution kernel operation on the target feature map data is determined to be met. If it is determined that the convolution kernel column index is the convolution column termination value, and the convolution kernel row index is lower than the total number of convolution kernel rows, then the convolution kernel column index is updated to the convolution column start value, the convolution kernel row index is incremented by one, and the image row index is incremented by one.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, perform the method as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method as described in any one of claims 1-3.

Images