Image processing method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment and a readable storage medium, and relates to the technical field of computers. In the image processing process, the pixel values corresponding to the regions of interest can be continuously stored in the cache in advance before the region features are determined, so that the pixel values of the pixel points in the regions of interest are continuously arranged, and therefore, the image processing efficiency is improved. Related data (namely data for determining regional characteristics) stored in the cache can be continuously arranged without being interfered by other data. Furthermore, when the regional features are determined, the continuous pixel value data can be read from the cache, the problems of interference of other data, cache miss and calculation redundancy are avoided, hardware resources are fully utilized, and the image processing performance and efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] At present, with the emergence and development of deep learning technology, electronic devices can process images based on deep learning technology to achieve purposes such as target detection.

[0003] In the process of image processing by related technologies, methods such as region of interest pooling can be used to share the parts with large computational workload and improve the efficiency of the model. However, related technologies still have problems such as insufficient utilization of hardware resources, computational redundancy, and cache misses, which results in low performance and efficiency of image processing. Summary of the invention

[0004] In view of this, embodiments of the present application provide an image processing method, apparatus, electronic device, and readable storage medium to improve the performance and efficiency of image processing.

[0005] In a first aspect, an image processing method is provided, the method comprising:

[0006] Determine the feature map corresponding to the target image.

[0007] A region coordinate set corresponding to each region of interest of the feature map is determined, wherein the region coordinate set includes a plurality of coordinate points for representing a range of the corresponding region of interest.

[0008] Each of the region coordinate sets is traversed, and the pixel values ​​of each pixel point in each of the regions of interest are continuously stored in a cache, so that the pixel values ​​of each pixel point in each of the regions of interest are continuously arranged.

[0009] The region feature corresponding to each of the regions of interest is determined according to each of the pixel values ​​stored in the buffer.

[0010] In some embodiments, the region of interest includes at least one sub-region, the region coordinate set also includes a plurality of coordinate points for representing the range of the sub-region, and each column of data in the cache is used to represent each pixel value corresponding to a sub-region.

[0011] Determining the region features corresponding to each of the regions of interest according to each of the pixel values ​​stored in the cache includes:

[0012] The maximum pixel value in each column direction in the cache is determined to determine the characteristic value corresponding to each of the sub-regions.

[0013] The region features corresponding to each of the regions of interest are determined according to the feature values ​​corresponding to each of the sub-regions.

[0014] In some embodiments, the region of interest includes at least one sub-region, and the region coordinate set also includes a plurality of coordinate points for characterizing the range of the sub-region.

[0015] Determining the region features corresponding to each of the regions of interest according to each of the pixel values ​​stored in the cache includes:

[0016] A pixel value of at least one sampling point in each of the sub-regions is determined.

[0017] According to the pixel value of each sampling point, the characteristic value corresponding to each sub-region is determined.

[0018] The region features corresponding to each of the regions of interest are determined according to the feature values ​​corresponding to each of the sub-regions.

[0019] In some embodiments, determining the pixel value of at least one sampling point in each of the sub-areas comprises:

[0020] For each sampling point, a predetermined number of pixel points closest to the sampling point are determined.

[0021] A weight matrix corresponding to the sampling point is determined according to the distance between the sampling point and each of the pixel points.

[0022] The pixel value corresponding to the sampling point is determined according to the weight matrix and a pixel value matrix composed of the pixel values ​​of each of the pixel points.

[0023] In some embodiments, determining the characteristic value corresponding to each of the sub-regions according to the pixel value of each of the sampling points includes:

[0024] For each of the sub-regions, an average value of pixel values ​​corresponding to the sampling points in the sub-region is determined.

[0025] The average value is used as the characteristic value corresponding to the sub-region.

[0026] In some embodiments, determining the characteristic value corresponding to each of the sub-regions according to the pixel value of each of the sampling points includes:

[0027] For each of the sub-regions, a maximum value of pixel values ​​corresponding to the sampling points in the sub-region is determined.

[0028] The maximum value is used as the characteristic value corresponding to the sub-region.

[0029] In a second aspect, an image processing device is provided, the device comprising:

[0030] The feature map acquisition module is configured to execute the acquisition of the feature map corresponding to the target image.

[0031] The region coordinate set determination module is configured to determine the region coordinate set corresponding to each interest region of the feature map, wherein the region coordinate set includes a plurality of coordinate points for representing the range of the corresponding interest region.

[0032] The pixel value storage module is configured to traverse each of the area coordinate sets and continuously store the pixel values ​​of each pixel point in each of the interest areas into a cache so that the pixel values ​​of each pixel point in each of the interest areas are continuously arranged.

[0033] The region feature determination module is configured to determine the region feature corresponding to each of the regions of interest according to each of the pixel values ​​stored in the cache.

[0034] In some embodiments, the region of interest includes at least one sub-region, the region coordinate set also includes a plurality of coordinate points for representing the range of the sub-region, and each column of data in the cache is used to represent each pixel value corresponding to a sub-region.

[0035] The region feature determination module is specifically configured to execute:

[0036] The maximum pixel value in each column direction in the cache is determined to determine the characteristic value corresponding to each of the sub-regions.

[0037] The region features corresponding to each of the regions of interest are determined according to the feature values ​​corresponding to each of the sub-regions.

[0038] In some embodiments, the region of interest includes at least one sub-region, and the region coordinate set also includes a plurality of coordinate points for characterizing the range of the sub-region.

[0039] The region feature determination module is specifically configured to execute:

[0040] A pixel value of at least one sampling point in each of the sub-regions is determined.

[0041] According to the pixel value of each sampling point, the characteristic value corresponding to each sub-region is determined.

[0042] The region features corresponding to each of the regions of interest are determined according to the feature values ​​corresponding to each of the sub-regions.

[0043] In some embodiments, the region feature determination module is specifically configured to execute:

[0044] For each sampling point, a predetermined number of pixel points closest to the sampling point are determined.

[0045] A weight matrix corresponding to the sampling point is determined according to the distance between the sampling point and each of the pixel points.

[0046] The pixel value corresponding to the sampling point is determined according to the weight matrix and a pixel value matrix composed of the pixel values ​​of each of the pixel points.

[0047] In some embodiments, the region feature determination module is specifically configured to execute:

[0048] For each of the sub-regions, an average value of pixel values ​​corresponding to the sampling points in the sub-region is determined.

[0049] The average value is used as the characteristic value corresponding to the sub-region.

[0050] In some embodiments, the region feature determination module is specifically configured to execute:

[0051] For each of the sub-regions, a maximum value of pixel values ​​corresponding to the sampling points in the sub-region is determined.

[0052] The maximum value is used as the characteristic value corresponding to the sub-region.

[0053] In a third aspect, an embodiment of the present application provides a chip for image processing, including a neural network processing unit.

[0054] The neural network processing unit includes multiple clusters, wherein each of the clusters includes multiple processing cores, each of the processing cores includes a computing unit and a cache, and the cache continuously stores pixel values ​​of at least one area of ​​interest of the feature map.

[0055] The calculation unit is used to obtain the pixel values ​​of the interest region from its cache to perform vector calculation or matrix calculation to determine the area features corresponding to the interest region.

[0056] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0057] In some embodiments, the processor includes a neural network processing unit, which includes multiple processing cores, each of which is used to execute one or more computer program instructions mentioned above to implement the method described in the first aspect.

[0058] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0059] In the process of image processing, since the embodiment of the present application can store the pixel values ​​corresponding to each region of interest in the cache before determining the regional features, so that the pixel values ​​of each pixel point in each region of interest are arranged continuously, the relevant data stored in the cache (i.e., the data used to determine the regional features) can be arranged continuously without being interfered with by other data. Furthermore, when determining the regional features, the embodiment of the present application can read continuous pixel value data from the cache, avoiding interference from other data, cache misses, and computational redundancy, making full use of hardware resources, and improving the performance and efficiency of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The above and other purposes, features and advantages of the embodiments of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0061] Figure 1 A schematic diagram of the flow of an image processing method according to an embodiment of the present application;

[0062] Figure 2 This is a schematic diagram of the structure of a neural network processing unit in a chip used for image processing in an embodiment of the present application;

[0063] Figure 3 A flowchart of an image processing method according to an embodiment of the present application;

[0064] Figure 4 A schematic diagram of an area of ​​interest, a sub-area, and area features of an embodiment of the present application;

[0065] Figure 5 A flowchart of another image processing method according to an embodiment of the present application;

[0066] Figure 6 A flowchart of another image processing method according to an embodiment of the present application;

[0067] Figure 7 A flowchart of another image processing method according to an embodiment of the present application;

[0068] Figure 8 A schematic diagram of a process for determining pixel values ​​corresponding to sampling points according to an embodiment of the present application;

[0069] Fig. 9 A schematic diagram of the structure of an image processing device according to an embodiment of the present application;

[0070] Fig.10 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0071] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0072] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0073] Unless the context clearly requires otherwise, the words "include", "comprising" and the like throughout this application should be interpreted as including rather than exclusive or exhaustive; that is, as meaning "including but not limited to".

[0074] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0075] With the emergence and development of deep learning technology, electronic devices can process images based on deep learning technology to achieve purposes such as target detection. Specifically, images can be processed through the Region Convolutional Neural Networks (R-CNN) model to detect targets appearing in the image. However, R-CNN has the disadvantages of cumbersome training, low efficiency, and large memory usage. Therefore, related technologies have improved R-CNN to obtain Fast R-CNN to improve the shortcomings of R-CNN.

[0076] Fast R-CNN is a fast recurrent neural network that can be used to process sequence data such as text, speech, and video. The main advantage of Fast R-CNN is that it can process a large amount of data in a shorter time and converge faster (that is, it has faster training efficiency). Specifically, Fast R-CNN can share the parts with larger computational workloads through the processing method of Region of Interest Pooling (RoI Pooling), thereby improving the efficiency of the model.

[0077] Furthermore, the processing method of RoI Pooling has certain defects, such as information loss and inaccurate positioning during the mapping process. In order to reduce these problems, the related technology improves the processing method of RoI Pooling to obtain RoI Align. Specifically, RoI Align can improve the precision and accuracy of feature extraction by performing bilinear interpolation on the region of interest (ie, ROI).

[0078] However, in the related technologies, whether it is RoI Pooling or RoI Align, there are still problems such as insufficient utilization of hardware resources, computational redundancy, and cache misses. Taking the Graphics Processing Unit (GPU) as an example, in the GPU architecture, there are a large number of streaming multiprocessor (SM) structures, and each SM structure includes a streaming processor (SP) structure. When the GPU performs image processing, the computing tasks can be assigned to each SP structure, and each SP structure can perform separate calculations based on RoI Pooling or RoI Align to achieve image processing.

[0079] In this process, affected by the GPU mechanism, the GPU has to use the "loop + if-else" code structure, which causes the same data to be loaded into multiple SP structures, resulting in a significant reduction in the performance and efficiency of image processing. Moreover, if different SP structures load the same data, these SP structures still need to distribute and perform computing tasks separately, resulting in computing redundancy in the image processing process, which reduces the performance and efficiency of image processing. In addition, if different SP structures access the same memory at the same time, it will cause memory access conflicts, which will also reduce the performance and efficiency of image processing. Therefore, how to effectively improve the performance and efficiency of image processing is a problem that needs to be solved urgently.

[0080] In order to solve the above problems, an embodiment of the present application provides an image processing method and a chip that can be used for image processing. The chip can be set in an electronic device for image processing. The image processing method of the embodiment of the present application can be applied to the electronic device and the above chip, wherein the electronic device can be a terminal or a server, the terminal can be a smart phone, a tablet computer, a vehicle control terminal or a personal computer (PC), etc., the server can be a single server, a server cluster configured in a distributed manner, or a cloud server.

[0081] like Figure 1As shown, the electronic device 12 can obtain the target image 11 and determine the regional features 13 according to the above image processing method. The target image 11 can be an image obtained by the electronic device 12 from an external device (such as a camera connected to the electronic device 12), or an image taken by the electronic device 12 based on its own shooting unit. The regional features 13 are used to characterize the features corresponding to each region of interest in the feature map corresponding to the target image 11.

[0082] Specifically, in the process of determining the regional features 13 according to the above-mentioned image processing method, the electronic device 12 can determine the feature map corresponding to the target image, and determine the regional coordinate set corresponding to each region of interest of the feature map. Further, the electronic device 12 can traverse each region coordinate set, and continuously store the pixel values ​​of each pixel point in each region of interest into the cache, so that the pixel values ​​of each pixel point in each region of interest are continuously arranged. Further, the electronic device 12 can determine the regional features 13 corresponding to each region of interest based on each pixel value stored in the cache. Among them, the regional coordinate set includes a plurality of coordinate points for characterizing the range of the corresponding region of interest.

[0083] In this process, since the embodiment of the present application can store the pixel values ​​corresponding to each region of interest in the cache before determining the regional features, so that the pixel values ​​of each pixel point in each region of interest are arranged continuously, the relevant data stored in the cache (i.e., the data used to determine the regional features) can be arranged continuously without being interfered with by other data. Furthermore, when determining the regional features, the embodiment of the present application can read continuous pixel value data from the cache, avoiding interference from other data, cache misses, and computational redundancy, making full use of hardware resources, and improving the performance and efficiency of image processing.

[0084] In addition, the chip for image processing of the embodiment of the present application may include a neural processor unit (NPU), and the neural processor unit may include multiple clusters, wherein each cluster may include multiple processing cores, and each processing core may include a computing unit and a cache, and the cache continuously stores pixel values ​​of at least one region of interest of the feature map. The computing unit can be used to obtain the pixel values ​​of the region of interest from its cache to perform vector calculation or matrix calculation to determine the regional features corresponding to the region of interest.

[0085] like Figure 2As shown, in the neural network processing unit 2, a total memory 21 and multiple clusters (i.e., cluster 22 to cluster 2n) may be included, where n is a natural number greater than or equal to 2. Each cluster may include multiple processing cores. Taking cluster 22 as an example, cluster 22 may include a shared memory 221 and multiple processing cores (i.e., processing core 222 to processing core 22m), where m is a natural number greater than or equal to 2. Each processing core may include a computing unit (e.g., Figure 2 The vector calculation unit and matrix calculation unit shown in the figure) and cache are used. Taking the processing core 222 as an example, the processing core 222 may include a calculation unit (such as a vector calculation unit 2221 and a matrix calculation unit 2222) and a cache 2223. The cache continuously stores pixel values ​​of at least one region of interest of the feature map; the calculation unit is used to obtain the pixel value of the region of interest from its cache to perform vector calculation or matrix calculation to determine the region feature corresponding to the region of interest.

[0086] The vector calculation unit can be used to perform vector calculation on the pixel values ​​of the region of interest, and the matrix calculation unit can be used to perform matrix calculation on the pixel values ​​of the region of interest, which can improve calculation efficiency.

[0087] In this embodiment, each processing core in the cluster can perform vector or matrix calculations on the pixel values ​​of multiple regions of interest, thereby improving the calculation efficiency. In addition, the cache of each processing core can be used to continuously store the pixel values ​​corresponding to the pixel points in each region of interest, that is, the pixel values ​​are arranged continuously, so that when determining the regional features, the processing core can directly read the continuous pixel value data from the cache of each processing core, avoiding interference from other data, cache misses and calculation redundancy, making full use of hardware resources, and improving the performance and efficiency of image processing.

[0088] In addition, for the electronic device of the embodiment of the present application, in an optional implementation, the processor of the electronic device may include a neural network processing unit (for example, the neural network processing unit may be Figure 2 The neural network processing unit 2) shown in the figure may include multiple processing cores, and each processing core may process multiple regional coordinate sets.

[0089] That is to say, the embodiment of the present application can set the neural network processing unit in the processor of the electronic device, so that when the electronic device determines the regional features, it can directly read continuous pixel value data from the cache of each processing core, avoiding interference from other data, cache misses and computational redundancy, making full use of hardware resources and improving the performance and efficiency of image processing.

[0090] The following will describe in detail an image processing method provided by an embodiment of the present application in combination with a specific implementation manner. Figure 3 As shown, the specific steps are as follows:

[0091] In step S100, a feature map corresponding to a target image is determined.

[0092] Among them, the feature map is an image obtained after the original image is feature mapped, and the image can be used to characterize a certain feature (color, shape, category, etc.) in the original image. In practical applications, feature maps are often used in scenes such as target recognition and target classification. In the embodiment of the present application, after obtaining the target image, the target image can be processed by convolution, pooling, etc. based on a pre-set network model to determine the feature map corresponding to the target image.

[0093] In step S200, a set of region coordinates corresponding to each region of interest (ROI) of the feature map is determined.

[0094] Among them, the region of interest is an area to be processed in the form of a box, circle, ellipse, irregular polygon, etc., which is outlined from the processed image in machine vision and image processing. In practical applications, the region of interest can be determined by a selective search algorithm. In an embodiment of the present application, the region coordinate set includes a plurality of coordinate points for characterizing the range of the corresponding region of interest, wherein the number of coordinate points in the region coordinate set is determined by the shape of the region of interest. Taking the region of interest in the form of a rectangular frame as an example, the range of the rectangular frame can be represented by at least 2 coordinate points (i.e., the coordinates of the two points forming diagonals in the rectangular frame). Therefore, the region coordinate set in this case can include at least 2 coordinate points, that is, the number of coordinate points in the region coordinate set in this case is greater than or equal to 2.

[0095] In step S300, each region coordinate set is traversed, and the pixel values ​​of each pixel point in each region of interest are continuously stored in a cache, so that the pixel values ​​of each pixel point in each region of interest are continuously arranged.

[0096] Specifically, the embodiments of the present application can determine the range identified by the corresponding region of interest based on the coordinate points in each region coordinate set, and further determine each pixel point in the region of interest and the pixel value corresponding to each pixel point.

[0097] Furthermore, in the embodiment of the present application, the pixel values ​​of each pixel point in each region of interest can be continuously stored in the cache in the order corresponding to the regions of interest, so that the pixel values ​​of each pixel point in each region of interest are arranged continuously. The order corresponding to the regions of interest can be the order in which the regions of interest are generated, can be a predetermined order, or can be an order generated according to other applicable rules.

[0098] For example, the embodiment of the present application can store the pixel value data corresponding to the region of interest or the sub-region in the region of interest as a column of data in the memory in sequence according to the above order, so that each column of data in the cache represents the pixel value data corresponding to the region of interest or a sub-region in the region of interest. Furthermore, when performing image processing, the embodiment of the present application can determine a maximum value in each column of data (i.e., perform maximum value pooling processing on each column of data) to determine the characteristic value of the region of interest or sub-region corresponding to the column of data.

[0099] In step S400, the region features corresponding to each region of interest are determined according to each pixel value stored in the buffer.

[0100] Specifically, the embodiment of the present application can determine the regional features corresponding to each region of interest by processing methods such as maximum pooling, bilinear difference, and average pooling, etc. The regional feature can be a feature value or a matrix composed of multiple feature values.

[0101] In this process, since the embodiment of the present application can store the pixel values ​​corresponding to each region of interest in the cache before determining the regional features, so that the pixel values ​​of each pixel point in each region of interest are arranged continuously, the relevant data stored in the cache (i.e., the data used to determine the regional features) can be arranged continuously without being interfered with by other data. Furthermore, when determining the regional features, the embodiment of the present application can read continuous pixel value data from the cache, avoiding interference from other data, cache misses, and computational redundancy, making full use of hardware resources, and improving the performance and efficiency of image processing.

[0102] Furthermore, after determining the regional features corresponding to each region of interest, the embodiments of the present application can use the regional features in various application scenarios such as target detection, target classification, and target recognition. The embodiments of the present application do not limit the scenarios in which the regional features are applied.

[0103] In an optional implementation, the region of interest may include at least one sub-region, and accordingly, the region coordinate set may also include multiple coordinate points for representing the range of the sub-region, and each column of data in the cache is used to represent each pixel value corresponding to a sub-region.

[0104] For example, Figure 4 As shown, the region of interest 41 may include four sub-regions (i.e., sub-region 411, sub-region 412, sub-region 413, and sub-region 414). When determining the regional features, the embodiment of the present application may determine a feature value for each sub-region, and combine the positions of the sub-regions and the feature values ​​corresponding to the sub-regions to obtain the regional features 42. Figure 4As shown, the eigenvalue corresponding to sub-region 411 is eigenvalue 421, the eigenvalue corresponding to sub-region 412 is eigenvalue 422, the eigenvalue corresponding to sub-region 413 is eigenvalue 423, and the eigenvalue corresponding to sub-region 414 is eigenvalue 424.

[0105] Furthermore, the embodiment of the present application can determine the regional features corresponding to the region of interest based on the maximum pooling processing method. In this case, Figure 5 As shown, the above step S400 may include the following steps:

[0106] In step S410, the maximum pixel value in each column direction in the buffer is determined to determine the feature value corresponding to each sub-region.

[0107] Specifically, the embodiment of the present application can determine a maximum value for the pixel value set corresponding to each sub-region (that is, each data in each column direction in the cache), and determine the maximum value as the characteristic value corresponding to each sub-region.

[0108] In an optional implementation, the embodiment of the present application can treat each column of data in the cache as a vector, and determine the maximum value in each column of data based on a vector processing instruction. Specifically, the vector processing instruction may include a "vemax_dimh()" instruction, which can be used to find the maximum value in the h direction (i.e., the column direction) of the matrix.

[0109] Through vector processing, when determining the eigenvalues ​​corresponding to each sub-region, the embodiment of the present application only requires one vector processing instruction to determine the eigenvalues ​​corresponding to the sub-region, which greatly improves the speed of determining the eigenvalues, thereby effectively improving the efficiency of image processing.

[0110] In step S420, the region features corresponding to each region of interest are determined according to the feature values ​​corresponding to each sub-region.

[0111] Specifically, the embodiment of the present application can combine the eigenvalues ​​of each sub-region according to the corresponding positions based on the positions of each sub-region and the eigenvalues ​​corresponding to each sub-region, so as to obtain the regional features corresponding to the region of interest. Since each sub-region corresponds to an eigenvalue, the number of eigenvalues ​​in the eigenvalue matrix used to characterize the regional features is the same as the number of sub-regions in the region of interest. For example, Figure 4 The region of interest 41 in includes 4 sub-regions, so the number of feature values ​​included in the region feature 42 corresponding to the region of interest 41 is also 4.

[0112] In an optional implementation, when the region of interest includes at least one sub-region, the embodiment of the present application may also determine the feature value corresponding to the sub-region by sampling, and then determine the region feature corresponding to the region of interest according to the feature value corresponding to the sub-region. Figure 6 As shown, the above step S400 may include the following steps:

[0113] In step S430, the pixel value of at least one sampling point in each sub-region is determined.

[0114] The sampling points in each sub-area may be randomly determined sampling points or sampling points determined in a specific manner. The number of sampling points in each sub-area may be one or more. The embodiment of the present application does not limit the number of sampling points.

[0115] In an optional implementation, since the coordinates of the sampling points are not integers, the sampling points in this case cannot completely correspond to the pixel points. In this case, the embodiment of the present application can determine the pixel values ​​of the sampling points by bilinear interpolation to improve the accuracy of the pixel values ​​corresponding to the sampling points. Specifically, Figure 7 As shown, step S430 may include the following steps:

[0116] In step S431, for each sampling point, a predetermined number of pixel points closest to the sampling point are determined.

[0117] The predetermined number may be set to an appropriate value according to actual conditions, and the value may be a natural number greater than or equal to 1.

[0118] In step S432, a weight matrix corresponding to the sampling point is determined according to the distance between the sampling point and each pixel point.

[0119] Among them, the embodiment of the present application can determine the sampling points corresponding to multiple sub-areas based on the same method. For the sampling points at the same position in the above multiple sub-areas, since the method of determining the sampling points is the same, the distances between the sampling points at the same position and their adjacent pixels are also the same, that is, in the channels for processing the sampling points at the same position, the same set of weights can be shared to achieve the purpose of fast processing.

[0120] For example, Figure 8 As shown, sub-region 81, sub-region 82 and sub-region 83 are three sub-regions in which sampling points are determined in the same manner, and each sub-region includes four sampling points (for example, sampling point 811, sampling point 812, sampling point 813 and sampling point 814 in sub-region 81). For each sampling point at the same position in these three sub-regions ( Figure 8 Taking the position shown by sampling point 811 as an example), the predetermined number closest to sampling point 811 ( Figure 8 For example, the predetermined number of pixels (4) is P1, P2, P3 and P4, and the distances between the sampling point 811 and the above four pixels are W1, W2, W3 and W4 respectively. Further, the embodiment of the present application can determine the weights corresponding to each pixel based on the distances between the sampling point 811 and the above four pixels, thereby determining the weight matrix corresponding to the sampling point 811. Among them, the embodiment of the present application can assign a larger weight to a pixel that is closer to the sampling point 811, and correspondingly, the embodiment of the present application can assign a smaller weight to a pixel that is farther away from the sampling point 811.

[0121] In an optional implementation, the embodiment of the present application may use the reciprocal of each distance as the weight of the corresponding pixel point. In this case, the weight corresponding to the sampling point 811 may be represented by the following matrix:

[0122]

[0123] The reciprocals of W1, W2, W3 and W4 are used to represent the weights corresponding to the pixels P1, P2, P3 and P4, respectively.

[0124] In another optional implementation, the embodiment of the present application may also use the distance between each pixel and the corresponding pixel as the weight corresponding to the pixel. Figure 8 For example, the pixel point opposite to the pixel point P1 is P3, and the pixel point opposite to the pixel point P2 is P4. In this case, the weight corresponding to the sampling point 811 can be represented by the following matrix:

[0125] [w 3 w 4 w 1 w 2 ]

[0126] Among them, W3 is the weight corresponding to pixel point P1, W4 is the weight corresponding to pixel point P2, W1 is the weight corresponding to pixel point P3, and W2 is the weight corresponding to pixel point P4. In other words, when the sampling point is closer to a certain pixel point, the sampling point will be further away from another relative pixel point accordingly. Figure 8For example, sampling point 811 is closer to pixel points P1 and P4, and correspondingly, sampling point 811 is farther from pixel points P1 and P4. At this time, the embodiment of the present application can use the distance corresponding to each pixel point relative to the pixel point as the weight corresponding to the pixel point (that is, the distance corresponding to pixel points P1 and P3 is used as the weight of the other party, and the distance corresponding to pixel points P2 and P4 is used as the weight of the other party). In this way, when the embodiment of the present application performs calculations based on weights, it can directly perform calculations based on distances without converting the distances (for example, without determining the inverse of the distance), which effectively improves the efficiency of determining the pixel values ​​corresponding to the sampling points, thereby improving the performance and efficiency of image processing.

[0127] It should be noted that the above implementation is an example of an embodiment of the present application. In practical applications, the embodiment of the present application may also determine the weight matrix in other applicable ways. For example, the embodiment of the present application may pre-set a mapping relationship between a distance value and a weight. After determining a predetermined number of pixel points closest to the sampling point, the embodiment of the present application may perform mapping according to the distance corresponding to each pixel point, determine the weight corresponding to each pixel point, and thus determine the weight matrix corresponding to the sampling point.

[0128] In step S433, the pixel value corresponding to the sampling point is determined according to the pixel value matrix composed of the weight matrix and the pixel value of each pixel point.

[0129] by Figure 8 For example, the pixel value matrix composed of the pixel values ​​of each pixel point can be represented as the following matrix:

[0130]

[0131] Need to explain, Figure 8 Three sub-areas are used as an example for description, and the embodiment of the present application does not limit the number of sub-areas.

[0132] Furthermore, since the embodiment of the present application can share the same set of weights in the channels for processing the sampling points at the same position, the process of determining the pixel value corresponding to the sampling point can be represented by the following formula:

[0133]

[0134] Among them, the above formula uses the distance corresponding to each pixel point relative to the pixel point as the weight corresponding to the pixel point, that is, W3 is the weight corresponding to pixel point P1, W4 is the weight corresponding to pixel point P2, W1 is the weight corresponding to pixel point P3, and W2 is the weight corresponding to pixel point P4.

[0135] After determining the weight matrix and the pixel value matrix, the embodiment of the present application can quickly calculate the pixel values ​​corresponding to the sampling points through the matrix multiplication instruction (memul_mm). Compared with conventional scalar calculations, the embodiment of the present application can effectively improve the efficiency of determining the pixel values ​​corresponding to the sampling points, thereby improving the performance and efficiency of image processing.

[0136] In addition, since the embodiment of the present application can store the pixel values ​​corresponding to each region of interest in advance in the cache continuously, so that the pixel values ​​of each pixel point in each region of interest are arranged continuously, and the data is usually continuous in the c direction (i.e., the channel direction) in the memory (that is, the pixels at the same position are continuous in the c direction), the embodiment of the present application can read the data in the memory sequentially to determine the pixel value matrix composed of the pixel values ​​of each pixel point. Figure 8 For example, pixel points P1, P5 and P9 are continuous in the c direction, pixel points P2, P6 and P10 are continuous in the c direction, pixel points P3, P7 and P11 are continuous in the c direction, and pixel points P4, P8 and P12 are continuous in the c direction.

[0137] That is to say, since the pixels at the same position are continuous in the c direction, the embodiment of the present application can use each pixel value in the c direction as a vector to participate in the matrix calculation, thereby simplifying the above formula to further improve the efficiency of determining the pixel value corresponding to the sampling point. Specifically, the simplified matrix multiplication operation can be represented by the following formula:

[0138]

[0139] in, It is used to represent the vector formed by the pixel point P1 in the c direction. It is used to represent the vector formed by the pixel point P2 in the c direction. It is used to represent the vector formed by the pixel point P3 in the c direction. Used to represent the vector formed by pixel point P4 in the c direction.

[0140] After determining the weight matrix and the pixel value matrix, the embodiment of the present application can quickly calculate the pixel values ​​corresponding to the sampling points through the matrix multiplication instruction (memul_mm). Compared with conventional scalar calculations, the embodiment of the present application can effectively improve the efficiency of determining the pixel values ​​corresponding to the sampling points, thereby improving the performance and efficiency of image processing.

[0141] In step S440, the feature value corresponding to each sub-region is determined according to the pixel value of each sampling point.

[0142] Among them, the embodiment of the present application can determine the characteristic value corresponding to each sub-region through average pooling processing, maximum pooling processing, etc.

[0143] In an optional implementation, the embodiment of the present application may determine the feature value corresponding to each sub-region by average pooling processing. Specifically, the above step S440 may include the following steps:

[0144] In step A1, for each sub-region, an average value of pixel values ​​corresponding to each sampling point in the sub-region is determined.

[0145] In step A2, the average value is taken as the feature value corresponding to the sub-region.

[0146] by Figure 8 For example, if the embodiment of the present application determines that the pixel values ​​corresponding to each sampling point in sub-region 81 (i.e., sampling point 811, sampling point 812, sampling point 813, and sampling point 814) are X1, X2, X3, and X4, respectively, then the characteristic value corresponding to sub-region 81 can be the average value of X1, X2, X3, and X4.

[0147] In the embodiment of the present application, the characteristic value corresponding to the sub-region can be accurately determined through bilinear interpolation and average pooling processing, thereby improving the accuracy of regional features.

[0148] In an optional implementation, the embodiment of the present application may determine the feature value corresponding to each sub-region by means of maximum pooling processing. Specifically, the above step S440 may include the following steps:

[0149] In step B1, for each sub-region, the maximum value of the pixel values ​​corresponding to the sampling points in the sub-region is determined.

[0150] In step B2, the maximum value is taken as the feature value corresponding to the sub-region.

[0151] by Figure 8 For example, if the embodiment of the present application determines that the pixel values ​​corresponding to each sampling point in sub-region 81 (i.e., sampling point 811, sampling point 812, sampling point 813, and sampling point 814) are X1, X2, X3, and X4, respectively, then the characteristic value corresponding to sub-region 81 can be the maximum value among X1, X2, X3, and X4.

[0152] In the embodiment of the present application, the speed of determining the characteristic value corresponding to the sub-region can be increased through bilinear interpolation and maximum pooling processing, thereby improving the efficiency of determining regional features.

[0153] In step S450, the region features corresponding to each region of interest are determined according to the feature values ​​corresponding to each sub-region.

[0154] In the process of determining regional features, since the embodiment of the present application can store the pixel values ​​corresponding to each region of interest in the cache before determining the regional features, so that the pixel values ​​of each pixel point in each region of interest are arranged continuously, the relevant data stored in the cache (i.e., the data used to determine the regional features) can be arranged continuously without being interfered with by other data. Furthermore, when determining regional features, the embodiment of the present application can read continuous pixel value data from the cache, avoiding interference from other data, cache misses, and computational redundancy, making full use of hardware resources, and improving the performance and efficiency of image processing.

[0155] Based on the same technical concept, the embodiment of the present application also provides an image processing device, such as Fig. 9 As shown, the device includes: a feature map acquisition module 91, a region coordinate set determination module 92, a pixel value storage module 93 and a region feature determination module 94.

[0156] The feature map acquisition module 91 is configured to acquire a feature map corresponding to a target image.

[0157] The region coordinate set determination module 92 is configured to determine a region coordinate set corresponding to each region of interest of the feature map, wherein the region coordinate set includes a plurality of coordinate points for representing a range of the corresponding region of interest.

[0158] The pixel value storage module 93 is configured to traverse each of the area coordinate sets and continuously store the pixel values ​​of each pixel point in each of the interest regions into a cache so that the pixel values ​​of each pixel point in each of the interest regions are continuously arranged.

[0159] The region feature determination module 94 is configured to determine the region feature corresponding to each of the regions of interest according to each of the pixel values ​​stored in the buffer.

[0160] In some embodiments, the region of interest includes at least one sub-region, the region coordinate set also includes a plurality of coordinate points for representing the range of the sub-region, and each column of data in the cache is used to represent each pixel value corresponding to a sub-region.

[0161] The region feature determination module 94 is specifically configured to execute:

[0162] The maximum pixel value in each column direction in the cache is determined to determine the characteristic value corresponding to each of the sub-regions.

[0163] The region features corresponding to each of the regions of interest are determined according to the feature values ​​corresponding to each of the sub-regions.

[0164] In some embodiments, the region of interest includes at least one sub-region, and the region coordinate set also includes a plurality of coordinate points for characterizing the range of the sub-region.

[0165] The region feature determination module 94 is specifically configured to execute:

[0166] A pixel value of at least one sampling point in each of the sub-regions is determined.

[0167] According to the pixel value of each sampling point, the characteristic value corresponding to each sub-region is determined.

[0168] The region features corresponding to each of the regions of interest are determined according to the feature values ​​corresponding to each of the sub-regions.

[0169] In some embodiments, the region feature determination module 94 is specifically configured to execute:

[0170] For each sampling point, a predetermined number of pixel points closest to the sampling point are determined.

[0171] A weight matrix corresponding to the sampling point is determined according to the distance between the sampling point and each of the pixel points.

[0172] The pixel value corresponding to the sampling point is determined according to the weight matrix and a pixel value matrix composed of the pixel values ​​of each of the pixel points.

[0173] In some embodiments, the region feature determination module 94 is specifically configured to execute:

[0174] For each of the sub-regions, an average value of pixel values ​​corresponding to the sampling points in the sub-region is determined.

[0175] The average value is used as the characteristic value corresponding to the sub-region.

[0176] In some embodiments, the region feature determination module 94 is specifically configured to execute:

[0177] For each of the sub-regions, a maximum value of pixel values ​​corresponding to the sampling points in the sub-region is determined.

[0178] The maximum value is used as the characteristic value corresponding to the sub-region.

[0179] In the process of image processing, since the embodiment of the present application can store the pixel values ​​corresponding to each region of interest in the cache before determining the regional features, so that the pixel values ​​of each pixel point in each region of interest are arranged continuously, the relevant data stored in the cache (i.e., the data used to determine the regional features) can be arranged continuously without being interfered with by other data. Furthermore, when determining the regional features, the embodiment of the present application can read continuous pixel value data from the cache, avoiding interference from other data, cache misses, and computational redundancy, making full use of hardware resources, and improving the performance and efficiency of image processing.

[0180] Fig.10 Schematic diagram of an electronic device according to an embodiment of the present application. Fig.10 As shown, Fig.10 The electronic device shown is a general address query device, which includes a general computer hardware structure, which at least includes a processor 101 and a memory 102. The processor 101 and the memory 102 are connected via a bus 103. The memory 102 is suitable for storing instructions or programs executable by the processor 101. The processor 101 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 101 executes the instructions stored in the memory 102, thereby executing the method flow of the embodiment of the present application as described above to realize the processing of data and the control of other devices. The bus 103 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 104 and the display device and the input / output (I / O) device 105. The input / output (I / O) device 105 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 105 is connected to the system via an input / output (I / O) controller 106.

[0181] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0183] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0184] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0185] Another embodiment of the present application relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0186] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0187] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Determine the feature map corresponding to the target image; Determine a region coordinate set corresponding to each region of interest of the feature map, wherein the region coordinate set includes a plurality of coordinate points for representing a range of the corresponding region of interest; Traversing each of the region coordinate sets, and continuously storing the pixel values ​​of each pixel point in each of the regions of interest into a cache, so that the pixel values ​​of each pixel point in each of the regions of interest are continuously arranged; The region feature corresponding to each of the regions of interest is determined according to each of the pixel values ​​stored in the buffer.

2. The method according to claim 1, characterized in that: The region of interest includes at least one sub-region, the region coordinate set also includes a plurality of coordinate points for representing the range of the sub-region, and each column of data in the cache is used to represent each pixel value corresponding to a sub-region; Determining the region features corresponding to each of the regions of interest according to each of the pixel values ​​stored in the cache includes: Determine the maximum pixel value in each column direction in the cache to determine the characteristic value corresponding to each sub-region; The region features corresponding to each of the regions of interest are determined according to the feature values ​​corresponding to each of the sub-regions.

3. The method according to claim 1, characterized in that The region of interest includes at least one sub-region, and the region coordinate set also includes a plurality of coordinate points for representing the range of the sub-region; Determining the region features corresponding to each of the regions of interest according to each of the pixel values ​​stored in the cache includes: Determine a pixel value of at least one sampling point in each of the sub-areas; Determining a characteristic value corresponding to each of the sub-regions according to the pixel value of each of the sampling points; The region features corresponding to each of the regions of interest are determined according to the feature values ​​corresponding to each of the sub-regions.

4. The method according to claim 3, characterized in that Determining the pixel value of at least one sampling point in each of the sub-areas comprises: For each sampling point, determining a predetermined number of pixel points closest to the sampling point; Determine a weight matrix corresponding to the sampling point according to the distance between the sampling point and each of the pixel points; The pixel value corresponding to the sampling point is determined according to the weight matrix and a pixel value matrix composed of the pixel values ​​of each of the pixel points.

5. The method according to claim 3, characterized in that: Determining the characteristic value corresponding to each of the sub-regions according to the pixel value of each of the sampling points includes: For each of the sub-regions, determining an average value of pixel values ​​corresponding to each sampling point in the sub-region; The average value is used as the characteristic value corresponding to the sub-region.

6. The method according to claim 3, characterized in that Determining the characteristic value corresponding to each of the sub-regions according to the pixel value of each of the sampling points includes: For each of the sub-regions, determining a maximum value of pixel values ​​corresponding to each sampling point in the sub-region; The maximum value is used as the characteristic value corresponding to the sub-region.

7. An image processing device, characterized in that: The device comprises: A feature map acquisition module is configured to acquire a feature map corresponding to a target image; A region coordinate set determination module is configured to determine a region coordinate set corresponding to each region of interest of the feature map, wherein the region coordinate set includes a plurality of coordinate points for representing a range of the corresponding region of interest; A pixel value storage module is configured to perform traversal of each of the area coordinate sets, and continuously store the pixel values ​​of each pixel point in each of the interest areas into a cache, so that the pixel values ​​of each pixel point in each of the interest areas are continuously arranged; The region feature determination module is configured to determine the region feature corresponding to each of the regions of interest according to each of the pixel values ​​stored in the cache.

8. A chip for image processing, comprising a neural network processing unit, characterized in that: The neural network processing unit includes a plurality of clusters, wherein each of the clusters includes a plurality of processing cores, each of the processing cores includes a computing unit and a cache, and the cache continuously stores pixel values ​​of at least one region of interest of the feature map; The calculation unit is used to obtain the pixel values ​​of the interest region from its cache to perform vector calculation or matrix calculation to determine the area features corresponding to the interest region.

9. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images