Image processing method, readable medium and electronic device

Abstract

The application relates to the technical field of artificial intelligence, and discloses an image processing method, a readable medium and an electronic device. The method comprises the following steps: acquiring a to-be-recognized image feature map; acquiring floating-point region position information of a region of interest on a to-be-recognized image, and quantizing the floating-point unit position information into fixed-point region position information; based on the fixed-point region position information of the region of interest, floating-point unit position information of a region of interest unit included in the region of interest is obtained; the floating-point unit position information is quantized to obtain fixed-point unit position information; based on the fixed-point unit position information, the feature values of the corresponding region of interest unit are acquired; based on the feature values of the region of interest unit in the region of interest, a region feature map of the region of interest is obtained, and a recognition result of the to-be-recognized image is obtained. By quantizing the position information of the region of interest in the to-be-recognized image and the position information of each unit of the region of interest, the running speed of a neural network model can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to an image processing method, readable medium and electronic device. BACKGROUND

[0002] With the development of artificial intelligence (AI) technology, deep learning takes a deep neural network (DNN) as a model, and has achieved very significant results in many key fields of artificial intelligence, such as image recognition, target detection, reinforcement learning, semantic analysis and the like.

[0003] Taking image recognition as an example, in the neural network model calculation process, the neural network model usually adopts floating point calculation for the input image to be processed. However, floating point calculation requires a large amount of calculation cost and occupies a large amount of memory, thereby affecting the running speed of the neural network model. In order to improve the running speed of the neural network model, it is necessary to quantize the neural network model to obtain a fixed-point operation neural network model, and then run the neural network model by an electronic device to realize the processing of the input image to be processed of the neural network model. SUMMARY

[0004] The present application aims to provide an image processing method, readable medium and electronic device.

[0005] The first aspect of the present application provides an image processing method, comprising: obtaining a feature map of an image to be recognized; obtaining floating point region position information of a region of interest on the image to be recognized, and quantizing the floating point unit position information into fixed point region position information; obtaining floating point unit position information of a unit of interest included in the region of interest based on the fixed point region position information of the region of interest; quantizing the floating point unit position information to obtain fixed point unit position information; obtaining a feature value of the unit of interest corresponding to the fixed point unit position information; obtaining a region feature map of the region of interest based on the feature values of the units of interest in the region of interest; and obtaining a recognition result of the image to be recognized based on the region feature map.

[0006] In a possible implementation of the first aspect, the floating point region position information includes position coordinates of a plurality of vertices of the region of interest.

[0007] In a possible implementation of the first aspect, obtaining the floating point region position information of the region of interest on the image to be recognized, and quantizing the floating point unit position information into fixed point region position information, comprises: obtaining the floating point region position information of the region of interest on the image to be recognized; determining a quantization parameter, and quantizing the floating point region position information based on the quantization parameter to obtain the fixed point region position information.

[0008] In a possible implementation of the first aspect, the floating-point region position information of the region of interest is obtained based on the fixed-point region position information of the region of interest, including: determining size information of a region feature map, and dividing the region of interest into a plurality of region units based on the size information of the region feature map and the fixed-point region position information; and determining the floating-point region position information of each region unit.

[0009] In a possible implementation of the first aspect, the size information of the region feature map is determined, and the region of interest is divided into a plurality of region units based on the size information of the region feature map and the fixed-point region position information, including: determining first size information of the region of interest and second size information of the region feature map, where the first size information includes a width and a height of the region of interest, and the second size information includes a width and a height of the region feature map; determining a height scaling coefficient based on the height of the region of interest and the height of the region feature map, and determining a width scaling coefficient based on the width of the region of interest and the width of the region feature map, where the height scaling coefficient represents a ratio of the height of the region of interest to the height of the region feature map, and the width scaling coefficient represents a ratio of the width of the region of interest to the width of the region feature map; and dividing the region of interest into a plurality of region units based on the height scaling coefficient and the width scaling coefficient, the fixed-point region position information of the region of interest, and the size information of the region feature map.

[0010] In a possible implementation of the first aspect, the region of interest is divided into a plurality of region units based on the height scaling coefficient and the width scaling coefficient, the fixed-point region position information of the region of interest, and the size information of the region feature map, including: determining a position of a top-left corner of the region of interest in the to-be-recognized image based on a position coordinate of a vertex of the top-left corner of the region of interest; determining height information and width information of each region unit based on the size information of the region feature map, the height scaling coefficient, and the width scaling coefficient; and determining floating-point region position information of each region unit in the to-be-recognized image based on the position of the top-left corner of the region of interest in the to-be-recognized image and the height information and the width information of each region unit, where the floating-point region position information includes position information of a plurality of boundaries of the region unit.

[0011] In a possible implementation of the first aspect, the height scaling factor is determined based on a height of the region of interest and a height of the region feature map, and the width scaling factor is determined based on a width of the region of interest and a width of the region feature map, including: respectively enlarging the height and the width of the region of interest by a preset ratio to obtain a first height and a first width; determining an integer division result of the first height and the height of the region feature map to obtain a height integer division result; determining an integer division result of the first width and the width of the region feature map to obtain a width integer division result; and respectively reducing the height integer division result and the width integer division result by the preset ratio to obtain the height scaling factor and the width scaling factor.

[0012] In a possible implementation of the first aspect, the floating-point unit position information is quantized to obtain the fixed-point unit position information, including: quantizing the floating-point unit position information based on a quantization parameter to obtain the fixed-point unit position information.

[0013] In a possible implementation of the first aspect, the feature value of the corresponding unit of interest is obtained based on the fixed-point unit position information, including: determining a pixel point in the to-be-recognized image corresponding to the fixed-point unit position information; obtaining a feature value in the feature map corresponding to the pixel point in each to-be-recognized image as a position feature value of the unit of interest; and determining a unit feature value of the unit of interest based on the fixed-point unit position information and the position feature value of the unit of interest.

[0014] In a possible implementation of the first aspect, the unit feature value of the unit of interest is determined based on the fixed-point unit position information and the position feature value of the unit of interest, including: determining a first interpolation coefficient and a second interpolation coefficient of each unit of interest based on the position feature value of the unit of interest; and determining the unit feature value of the unit of interest based on the first interpolation coefficient, the second interpolation coefficient, and the position feature value of the unit of interest.

[0015] In a possible implementation of the first aspect, the first interpolation coefficient and the second interpolation coefficient of each unit of interest are determined based on the position feature value of the unit of interest, including: determining a bit number n of a binary number corresponding to the fixed-point unit position information; wherein the bit number of the binary number corresponding to the fixed-point unit position information is equal to a bit number of a binary number corresponding to the fixed-point region position information; determining the last n bits of a binary number corresponding to position information of a left boundary of the unit of interest as the first interpolation coefficient; and determining the last n bits of a binary number corresponding to position information of an upper boundary of the unit of interest as the second interpolation coefficient.

[0016] The second aspect of the present application provides a readable medium, and the readable medium stores instructions. When the instructions are executed on an electronic device, the electronic device performs any one of the methods in the first aspect.

[0017] The third aspect of the present application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the processors of the electronic device, for executing any of the methods in the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings required to be used in the specific embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0019] Figure 1a According to the embodiments of the present application, a structural diagram of a target detection model is shown;

[0020] Figure 1b According to the embodiments of the present application, a schematic diagram of an image recognition process is shown;

[0021] Figure 2a According to the embodiments of the present application, a schematic diagram of a region of interest of a to-be-recognized image and the to-be-recognized image is shown;

[0022] Figure 2b According to the embodiments of the present application, a schematic diagram of a feature map of a to-be-recognized image is shown;

[0023] Figure 3a According to the embodiments of the present application, a schematic diagram of a method for determining a region feature map of a region of interest on a to-be-recognized image is shown;

[0024] Figure 3b According to the embodiments of the present application, a schematic diagram of a feature map and a region of interest on the feature map is shown;

[0025] Figure 3c According to the embodiments of the present application, a schematic diagram of a feature map and a region of interest on the feature map is shown;

[0026] Figure 3d According to the embodiments of the present application, a schematic diagram of a region of interest divided into multiple units is shown;

[0027] Figure 3e According to the embodiments of the present application, another schematic diagram of a region of interest divided into multiple units is shown;

[0028] Figure 4a According to the embodiments of the present application, a schematic diagram of a method for determining a region feature map of a region of interest on a to-be-recognized image is shown;

[0029] Figure 4b According to an embodiment of the present application, a schematic diagram of dividing a region of interest on a to-be-recognized image into a plurality of units is shown;

[0030] Figure 4c According to an embodiment of the present application, a schematic diagram of bilinear interpolation is shown;

[0031] Figure 5 According to an embodiment of the present application, a schematic diagram of an implementation process of an image processing method is shown;

[0032] Figure 6a According to an embodiment of the present application, a schematic diagram of a to-be-recognized image is shown;

[0033] Figure 6b According to an embodiment of the present application, a schematic diagram of a feature map of a to-be-recognized image is shown;

[0034] Figure 6c According to an embodiment of the present application, a schematic diagram of quantized position information of a top-left corner vertex of a region of interest is shown;

[0035] Figure 6d According to an embodiment of the present application, a schematic diagram of quantized position information of a top-left corner vertex and a left-top corner vertex of each unit, and quantized position information of an upper boundary and quantized position information of a left boundary of a first unit are shown;

[0036] Figure 6e According to an embodiment of the present application, a schematic diagram of x-direction interpolation coefficients and y-direction interpolation coefficients is shown;

[0037] Figure 7 According to some embodiments of the present application, a structural schematic diagram of an electronic device 100 is shown. DETAILED DESCRIPTION

[0038] Illustrative embodiments of the present application include, but are not limited to, an image processing method, a readable medium, and an electronic device.

[0039] As mentioned before, in the process of neural network model calculation, the neural network model usually adopts floating point calculation on the input to-be-processed image. However, floating point calculation requires a large amount of calculation cost and occupies a large amount of memory, thereby affecting the running speed of the neural network model. In order to improve the running speed of the neural network model, it is usually necessary to quantize the neural network model to obtain a fixed-point operation neural network model, and then run it by an electronic device to realize the processing of the input to-be-processed image of the neural network model.

[0040] For example, taking a target detection model 10 as an example, Figure 1a A structural schematic diagram of a target detection model is shown. As shown inFigure 1a As shown in the target detection model 10 includes a backbone network 11, a region proposal network 12 and a head network of a region of interest 13. Among them, the backbone network 11 extracts features of the input image through a plurality of convolution kernels to obtain a plurality of feature maps; the region proposal network 12 is used to generate a candidate bounding box corresponding to the output feature map of the backbone network 11, and adjust the candidate bounding box based on the output feature map to obtain a region of interest (ROI); the head network of the region of interest 13 is used to complete feature pooling and classification regression of the region of interest, and obtain image processing results and calculation loss.

[0041] Specifically, the following is based on the target detection model shown in the image recognition process of the neural network model is described in detail. Figure 1a As shown in the target detection model 10 includes a backbone network 11, a region proposal network 12 and a head network of a region of interest 13. Among them, the backbone network 11 extracts features of the input image through a plurality of convolution kernels to obtain a plurality of feature maps; the region proposal network 12 is used to generate a candidate bounding box corresponding to the output feature map of the backbone network 11, and adjust the candidate bounding box based on the output feature map to obtain a region of interest (ROI); the head network of the region of interest 13 is used to complete feature pooling and classification regression of the region of interest, and obtain image processing results and calculation loss.

[0042] Specifically, Figure 1b A schematic diagram of an image recognition process is shown. Specifically, as Figure 1b As shown in the target detection model 10 includes a backbone network 11, a region proposal network 12 and a head network of a region of interest 13. Among them, the backbone network 11 extracts features of the input image through a plurality of convolution kernels to obtain a plurality of feature maps; the region proposal network 12 is used to generate a candidate bounding box corresponding to the output feature map of the backbone network 11, and adjust the candidate bounding box based on the output feature map to obtain a region of interest (ROI); the head network of the region of interest 13 is used to complete feature pooling and classification regression of the region of interest, and obtain image processing results and calculation loss.

[0043] S101: Obtain an image to be recognized.

[0044] Specifically, the backbone network 11 obtains the image to be recognized input into the neural network model. For example, referring to the to-be-processed image S in Figure 2a , the height and width of the to-be-processed image S are both 512 (i.e., the to-be-processed image includes 512x512 pixel points).

[0045] S102: Generate a feature map.

[0046] Specifically, the backbone network 11 extracts features of the image to be recognized through a plurality of convolution kernels to obtain a plurality of feature maps of a plurality of channels of the image to be recognized, for example, a feature map of three channels of RGB of a color picture. For example, referring to the feature map F in Figure 2b , the height and width of the feature map F are both 16 (i.e., the feature map F includes 16x16 feature values / feature vectors).

[0047] S103: Determine a region of interest on the image to be recognized.

[0048] Specifically, the region proposal network 12 obtains a candidate bounding box generated according to the feature map output by the backbone network 11, judges whether the candidate bounding box belongs to the foreground or the background through a softmax function, and then obtains the region of interest on the image to be recognized by using a proposal operator.

[0049] For example, referring to Figure 2a , Figure 2aThe region of interest B1 on the image to be recognized S is shown, and the height of the region of interest B1 is 168.5 and the width is 200.5.

[0050] S104: Determine the region feature map of the region of interest on the image to be recognized.

[0051] Specifically, the head network 13 of the region of interest determines the region feature map of the region of interest on the image to be recognized based on the region of interest on the image to be recognized and the feature map.

[0052] S105: Recognize the image to be processed.

[0053] Specifically, the head network 13 of the region of interest determines the class and position information of the target object in the region of interest on the image to be recognized based on the region feature map.

[0054] In some embodiments, the method for determining the region feature map of the region of interest on the image to be recognized in the above S104 step can refer to Figure 3a . Specifically, Figure 3a A schematic diagram of a method for determining the region feature map of the region of interest on the image to be recognized is shown. Specifically, as shown in Figure 3a , the method comprises:

[0055] S301: Determine the region of interest on the feature map.

[0056] Specifically, the region of interest on the image to be recognized is mapped to the feature map to obtain the region of interest on the feature map. For example, Figure 3b The feature map F of the image to be recognized S shown in Figure 2a is shown, and as shown in Figure 3b , the corresponding region of interest B1 of the image to be recognized S shown in Figure 2a is the region of interest B2 on the feature map F, wherein the height of the region of interest B2 is 5.265625 and the width is 6.265625.

[0057] S302: Quantize the bounding box of the region of interest on the feature map.

[0058] Specifically, in some embodiments, the position information (top-left corner vertex and top-right corner vertex) of the region of interest can be quantized, for example, the position information of the region of interest B2 (height 5.265625, width 6.265625) on the feature map F shown in Figure 3b is quantized to the position information of the region of interest B3 shown in Figure 3c , to obtain a region of interest B3 with a height of 5 and a width of 6.

[0059] S303: Divide the region of interest on the feature map into units of interest.

[0060] Since the size of the region of interest in the image to be identified is determined based on the size of the target object in the region of interest, but the size of the region feature map of the region of interest input into the fully connected layer of the target detection model needs to be simplified to the size of the target feature map, in order to make the size of the feature map of each region of interest the same as the size of the target feature map, all regions of interest are divided into smaller units of interest of uniform size according to the size of the region feature map. The number of units of interest in each region of interest is the same as the number of feature values ​​in the region feature map.

[0061] For example, taking a region of interest as an example, for Figure 3c The feature map F of the image to be identified shows a region of interest (ROI) B3 with dimensions of height 5 and width 6, and a region feature map with dimensions of height 2 and width 2. Therefore, ROI B3 is divided into 4 (2×2) units of interest. For example... Figure 3d It shows that Figure 3c The diagram shows the region of interest B3 divided into 4 units, each with a width of 3 and a height of 2.5.

[0062] S304: Quantize the boundaries of each unit of interest.

[0063] In order to ensure that the position information of the vertices (top left, bottom left, top right, and bottom right) of each unit of interest corresponds to the position information of the feature values ​​in the feature map of the image to be processed, the boundaries of each unit of interest need to be quantized so that the position information of the vertices of each unit of interest corresponds to the position information of the feature values ​​in the feature map of the image to be processed.

[0064] For example, Figure 3e The pair shown Figure 3d The diagram shows the quantization results of the boundaries of the four units of interest, as shown below. Figure 3e As shown, for Figure 3d The boundaries of each cell of interest with a width of 3 and a height of 2.5 are quantized to obtain four quantized cells of interest with the following dimensions: height 2, width 3; height 2, width 3; height 3, width 3; height 3, width 3.

[0065] S305: Determine the feature values ​​of each unit of interest through pooling.

[0066] It can be understood that, in order to obtain the feature values of each unit of interest, the maximum value / average value of the feature values in each unit of interest of each region of interest in the feature map of the to-be-identified image can be determined by maximum pooling or average pooling, as the feature values of each unit of interest of each region of interest in the feature map of the to-be-identified image, and the feature values of each unit of interest of each region of interest are mapped to the feature values at the corresponding positions in the region feature map.

[0067] For example, Figure 3e An example of a method for obtaining a region feature map is shown in FIG. 6. Figure 3e An example of a mapping relationship between the feature values in each quantized unit of interest in the region of interest B3 and the feature values in the region feature map T is shown in FIG. 7. Figure 3e As shown in FIG. 7, the maximum feature values in each unit of interest in the region of interest B3 are mapped to the feature values at the corresponding positions in the region feature map T.

[0068] However, in the above method, the bounding box of the region of interest on the feature map is quantized once in the process of calculating the feature values / feature vectors of each unit of interest of the region of interest, and the boundaries of each unit of interest of each region of interest after quantization are quantized once, and the deviation of the region of interest obtained by twice quantization on the feature map from the position of the region of interest in the to-be-identified image input into the neural network model is large, which may affect the accuracy of the target feature obtained finally, and further affect the recognition result of the target object in the region of interest in the to-be-identified image.

[0069] To solve the above problem, in some embodiments of the present application, after the region of interest and the floating-point region position information of the region of interest are determined in the to-be-identified image, the region of interest on the to-be-identified image is directly divided into multiple units of interest on the to-be-identified image instead of the feature map of the to-be-identified image. Then, the feature values of the pixel points corresponding to the floating-point unit position information (such as the coordinate information of the top-left corner vertex, the bottom-left corner vertex, the top-right corner vertex, and the bottom-right corner vertex) of each unit of interest are confirmed from the feature map of the to-be-identified image (hereinafter referred to as position feature values). Then, based on the position feature values of each unit of interest, the feature values of each unit of interest (hereinafter referred to as unit feature values) are fitted. Thus, the region feature map of the region of interest of the to-be-identified image is obtained.

[0070] It can be understood that, since the bilinear interpolation method fits the unit feature values of each unit of interest by the feature values of multiple pixel points in the to-be-identified image, the fitted feature values are not the feature values of a certain pixel point, and can better reflect the image features of the unit of interest, so the unit feature values of the unit of interest can be fitted, thereby improving the accuracy of the unit feature values of the unit of interest.

[0071] In the method, after the region of interest and the floating point region position information of the region of interest in the image to be recognized are determined directly, the region of interest on the image to be recognized is divided into a plurality of interest units, the floating point unit position information of each interest unit is determined, the unit feature value of the interest unit is determined based on the floating point operation, and the deviation of the position of the region of interest caused by twice quantization on the feature map is avoided.

[0072] For example, Figure 4a A schematic diagram of a method for determining a region feature map of a region of interest on an image to be recognized is shown. Figure 4a The method shown improves the method for determining a region feature map of a region of interest on an image to be recognized in the S104 step. Specifically, as shown Figure 4a The method shown includes:

[0073] S401: Divide the region of interest on the image to be recognized into a plurality of units.

[0074] For example, in some embodiments of the present application, according to the size information (width w and height h) of the region feature map, the region of interest on the image to be recognized is divided into N (w x h) interest units. For example, taking one region of interest as an example, Figure 4b A schematic diagram of dividing Figure 2a The interest unit B1 shown is divided into 4 units, as shown Figure 4b As shown, the width of each interest unit is 50.125, and the height is 42.125.

[0075] S402: Determine the position information of the 4 vertices of each interest unit, and obtain the feature values corresponding to the 4 vertices.

[0076] For example, in some embodiments of the present application, the position information of the 4 vertices of each interest unit is determined, and the feature values corresponding to the pixel points corresponding to the position information of the 4 vertices of each interest unit in the feature map of the image to be processed are found as the feature values of the 4 vertices of each interest unit.

[0077] S403: Determine the target feature corresponding to each interest unit based on the position information of the 4 vertices of each interest unit and the corresponding feature values.

[0078] For example, in some embodiments of the present application, based on the feature values of the 4 vertices of each interest unit and the position information of each interest unit, the feature values of each interest unit are fitted according to the method of bilinear interpolation, the fitted feature values are taken as the feature values of each interest unit, and the fitted feature values are mapped as the target features of the corresponding positions of the region feature map.

[0079] For example, taking one unit of interest as an example, Figure 4c A schematic diagram of bilinear interpolation is shown. Q 11 , Q 12 , Q 21 , Q 22 represent the four vertices of the unit of interest, wherein Q 11 has coordinates (x1, y1), Q 12 has coordinates (x1, y2), Q 21 has coordinates (x2, y1), and Q 22 has coordinates (x2, y2), R1 and R2 represent interpolation points, R1 has coordinates (x, y1), and R2 has coordinates (x, y2), P represents a target point, i.e. a point corresponding to the characteristic value fitted for each unit of interest (which can be a center point, for example), and P has coordinates (x, y). The characteristic value corresponding to the P point can be determined by formula (1).

[0080] f(x, y) = Q12*(1-x lerp )*(1-y lerp )+Q22*x lerp *(1-y lerp )+Q11*(1-x lerp )*y lerp Q21*x lerp *y lerp (1)

[0081] wherein f(x, y) represents the characteristic value corresponding to the point P, Q11 is the characteristic value corresponding to the point Q 11 , Q12 is the characteristic value corresponding to the point Q 12 , Q21 is the characteristic value corresponding to the point Q 21 , and Q22 is the characteristic value corresponding to the point Q 22 . represents the ratio of the distance of the point P and the nearest pixel point in the x direction to the distance between the upper and lower two pixel points (x direction interpolation coefficient), and y lerp represents the ratio of the distance of the point P and the nearest pixel point in the y direction to the distance between the left and right two pixel points (y direction interpolation coefficient).

[0082] In the above method, the region of interest in the to-be-identified image is directly divided according to the size of the region feature map to obtain a plurality of units, and the feature values of the units are fitted to obtain the feature values of the corresponding positions of the region feature map, thereby avoiding the deviation of the position of the region of interest caused by twice quantization on the feature map. However, in the calculation process of the above method, floating point calculation is used, for example, the number of bits of the binary data of each data is 32 bits, and a large amount of calculation cost is still consumed, a large amount of hardware resources of the NPU is occupied, and the speed of the electronic device running the neural network model is affected; and the method is not suitable for the scene of running the neural network model to process the image by the NPU which does not support floating point operation.

[0083] To solve the above problems, an image processing method is provided in another embodiment of the present application, which can be used in the process of image recognition by the method as shown in Figure 4a The floating point region position information of the region of interest in the to-be-identified image and the floating point unit position information of the unit of interest in the region of interest are quantized to obtain the fixed point region position information of the region of interest and the fixed point unit position information of the unit of interest in the region of interest, and then subsequent processing is performed. Therefore, the unit feature values of each unit of interest are determined by the region feature map through fixed point operation, and the memory space can be reduced in the process of fixed point operation, thereby improving the running speed of the operation process.

[0084] For example, the position information of the region of interest B1 in the to-be-identified image S as shown in Figure 2a is quantized, for example, the binary bit number of the height and width of the region of interest B1 is determined to be 16 bits. According to the size information of the region feature map, the quantized region of interest B1 is divided to obtain a plurality of units of interest, for example Figure 4b The region of interest B1 is divided to obtain four units of interest as shown in The position information (the position information of each vertex) of each unit of interest is quantized, for example, the binary bit number of the height and width of each unit of interest is determined to be 16 bits. The fixed point position information corresponding to the feature values of each vertex of each unit of interest quantized from the feature map of the to-be-processed image is obtained, and the unit feature values of each unit of interest are determined according to the method of bilinear interpolation.

[0085] It can be understood that by quantizing the position information of the corresponding region of interest in the region feature map and quantizing the position information of each unit of interest, for example, the number of bits of the binary data of each data is determined to be 16 bits, the memory space is reduced compared with floating point calculation, and the running speed of the operation process can be improved.

[0086] In some embodiments, the electronic device determines floating-point region position information of the region of interest; determines a quantization parameter, and quantizes the floating-point region position information of the region of interest based on the quantization parameter to obtain fixed-point region position information; determines size information of the region feature map, and divides the region of interest into a plurality of units based on the size information of the region feature map and the fixed-point region position information; determines fixed-point unit position information (for example, position information of left, right, top and bottom boundaries) of each unit, and obtains feature values in the feature map corresponding to the fixed-point unit position information of each unit as position feature values; and determines unit feature values of each unit based on the fixed-point unit position information of each unit and the position feature values.

[0087] In some embodiments, the electronic device determines image size information to be processed, determines size information of the region of interest based on the image size information to be processed and the quantized position information of the region of interest; determines size information of the region feature map, and determines a height scaling factor and a width scaling factor based on the size information of the region of interest and the size information of the region feature map; divides the region of interest into N units based on the height scaling factor and the width scaling factor, the quantized position information of the region of interest and the size information of the region feature map; and N is equal to the number of target features in the region feature map.

[0088] In some embodiments, the electronic device respectively determines fixed-point unit position information of the upper, lower, left and right boundaries of each unit; determines fixed-point position information of the top-left corner vertex, the bottom-left corner vertex, the top-right corner vertex and the bottom-right corner vertex of each unit based on the fixed-point unit position information of the upper, lower, left and right boundaries of each unit; and if the fixed-point position information corresponding to the top-left corner vertex, the bottom-left corner vertex, the top-right corner vertex and the bottom-right corner vertex is not the position information of a pixel point in the image to be recognized, then the corresponding fixed-point unit position information is rounded according to rounding rules, so that the corresponding position information is the position information of a pixel point in the image to be recognized.

[0089] In some embodiments, the electronic device determines an interpolation coefficient based on the fixed-point unit position information of each unit, wherein the interpolation coefficient includes a first interpolation coefficient (x-direction interpolation coefficient) and a second interpolation coefficient (y-direction interpolation coefficient).

[0090] In some embodiments, the electronic device obtains position feature values in the feature map corresponding to the fixed-point unit position information of each unit, and determines unit feature values of each unit based on the position feature values and the interpolation coefficient.

[0091] It can be understood that the image processing method provided in the embodiments of the present application can be applied to any electronic device capable of running a neural network model, including but not limited to a mobile phone, a wearable device (such as a smart watch and the like), a tablet computer, a desktop computer, a laptop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) / virtual reality (VR) device, and the like. The embodiments of the present application do not make any limitation on the specific type of the electronic device.

[0092] It can be understood that the target detection model is taken as an example of the model to which the image is input for identification in the present application. It can be understood that other models capable of processing image data, such as a convolutional neural network (CNN), a recursive neural network (RNN), and the like, can also be applicable to the technical solutions of the embodiments of the present application.

[0093] In order to more clearly illustrate the technical solutions provided in the embodiments of the present application, the technical solutions of the present application will be described below in conjunction with the accompanying drawings.

[0094] Figure 5 According to the embodiments of the present application, an implementation flowchart of an image processing method is shown. It can be understood that, Figure 5 The execution subject of each step of the flowchart is an electronic device 100. In order to simplify the description, the execution subject of each step will not be repeatedly described below when introducing Figure 5 the flowchart. For example, Figure 5 The flowchart includes but is not limited to the following steps:

[0095] S501: Obtain a to-be-identified image, and perform feature extraction on the to-be-identified image to obtain a feature map.

[0096] In some embodiments, the electronic device 100 determines the size information of the to-be-identified image input into the neural network model, determines a scaling step of the feature map, and performs feature extraction on the to-be-processed image through a plurality of convolution kernels to obtain the feature map of the to-be-identified image, wherein the ratio of the size of the feature map to the size information of the to-be-identified image is the scaling step.

[0097] For example, with reference to Figure 6a , Figure 6aA schematic diagram of an image to be identified is shown. The image S has a width and height of 128, meaning it comprises 128 × 128 pixels. Assuming the feature map scaling step is 8, refer to... Figure 6b , Figure 6b A schematic diagram of a feature map of an image to be identified is shown. The feature map F has a width and height of 16, meaning that the feature map F includes 16×16 feature values.

[0098] S502: Determine the region of interest and the location information of the fixed point region of interest on the image to be identified.

[0099] In some embodiments, the electronic device 100 determines the region of interest (ROI) in the image to be recognized and the floating-point location information of the ROI, such as the coordinates of the top-left and bottom-right vertices of the ROI. It can be understood that the floating-point location information of the ROI is data in the floating-point range of 0 to 1.0 output by the proposal operator.

[0100] In some embodiments, the electronic device 100 quantizes the floating-point region position information (e.g., the coordinates of the top left and bottom right vertices) of the region of interest based on quantization parameters to obtain fixed-point region position information and determine the position information of the region of interest in the image to be recognized.

[0101] For example, in some embodiments, the floating-point region location information of the region of interest is a normalized parameter output by the region proposal network, whose floating-point range is from 0 to 1.0. Electronic device 100 determines the quantization parameter scale to be 2. 15 Based on the quantization parameter scale, the floating-point region location information of the region of interest is converted into 16-bit fixed-point region location information.

[0102] For example, the fixed-point region location information of the upper left vertex of the region of interest is (x0_q, y0_q), and the fixed-point region location information of the lower right vertex of the region of interest is (x1_q, y1_q).

[0103] The position information of the top left vertex of the region of interest in the image to be recognized is (x0_q*(image_width-1), y0_q*(image_height-1)), and the position information of the bottom right vertex of the region of interest in the image to be recognized is (x1_q*(image_width-1), y1_q*(image_height-1)), where image_width represents the width of the image to be recognized and image_height represents the height of the image to be recognized.

[0104] For example, Figure 6cThe fixed-point region position information of the top-left corner vertex of the region of interest B1 is (x0_q, y0_q), and the fixed-point region position information of the bottom-right corner vertex of the region of interest is (x1_q, y1_q).

[0105] S503: Determine the size information of the region feature map, and divide the region of interest into a plurality of units of interest based on the size information of the region feature map and the fixed-point region position information.

[0106] In some embodiments, the electronic device 100 determines the size information (height and width) of the region feature map, and divides the region of interest into N units of interest based on the size information of the region feature map and the fixed-point region position information.

[0107] In some embodiments, the electronic device 100 determines the height scaling factor (height_scale_q) and the width scaling factor (width_scale_q) according to the size information of the region of interest in the image to be recognized and the width (resize_width) and height (resize_height) of the region feature map. Then, the electronic device 100 divides the region of interest into N units of interest and determines the fixed-point unit position information (coordinates of the top-left corner vertex) of each unit of interest based on the height scaling factor and the width scaling factor, the height and width of the region feature map, and the position information of the region of interest in the image to be recognized. It can be understood that N=image_height*resize_width. For example Figure 6c The size information of the region of interest is shown as height (y1_q-y0_q)*(image_height-1) and width (x1_q-x0_q)*(image_width-1).

[0108] In some embodiments, the height scaling factor height_scale_q can be determined by the height of the region of interest and the height of the region feature map. By expanding the height of the region of interest in the image to be recognized by a preset ratio, for example, multiplying by 2 8 (or, performing a left shift operation on the binary number of the height of the region of interest by 8 bits), and then performing an integer division operation on the height of the region feature map, and dividing the result of the integer division by 2 8 (or, performing a right shift operation on the binary number of the result of the integer division by 8 bits).

[0109] It can be understood that multiplying the height of the region of interest in the image to be recognized by 2 8 and then dividing the result of the integer division by 2 8 does not change the number of bits of the height scaling factor, and in this way the accuracy of the height scaling factor can be improved.

[0110] Exemplarily, the height scaling coefficient height_scale_q can be determined by the following code.

[0111] height_scale_q = ((y1_q - y0_q) * (image_height - 1) * 256 / (resize_height - 1)) >> 8

[0112] wherein (y1_q - y0_q) * (image_height - 1) represents the height of the region of interest, y1_q represents the fixed-point region position information in the y direction of the bottom-right corner vertex of the region of interest, y0_q represents the fixed-point region position information in the y direction of the top-left corner vertex of the region of interest, image_height represents the height of the image to be recognized, and resize_height represents the height of the region feature map.

[0113] In some embodiments, the width scaling coefficient width_scale_q can be determined by the width of the region of interest and the width of the region feature map. By expanding the width of the region of interest in the image to be recognized by a predetermined proportion, for example, multiplying by 2 8 (or, performing a left shift operation on the binary number of the width of the region of interest by 8 bits), and then performing an integer division operation on the width of the region feature map, and then dividing the result of the integer division by 2 8 (or, performing a right shift operation on the binary number of the result of the integer division by 8 bits).

[0114] It can be understood that, after multiplying the width of the region of interest in the image to be recognized by 2 8 , and then dividing the result of the integer division by 2 8 , the number of bits of the width scaling coefficient is not changed, and in this way the precision of the width scaling coefficient can be improved.

[0115] Exemplarily, the width scaling coefficient width_scale_q can be determined by the following code.

[0116] width_scale_q = ((x1_q - x0_q) * (image_width - 1) * 256 / (resize_width - 1)) >> 8

[0117] wherein (x1_q - x0_q) * (image_width - 1) represents the width of the region of interest, x1_q represents the fixed-point region position information in the x direction of the bottom-right corner vertex of the region of interest, x0_q represents the fixed-point region position information in the x direction of the top-left corner vertex of the region of interest, image_width represents the width of the image to be recognized, and resize_width represents the width of the region feature map.

[0118] In some embodiments, the fixed-point cell position information of the top-left vertex of each of the N interest units can be determined by the following code.

[0119] x_q = (image_width - 1) * x0_q + torch.arange(0, resize_width, device=out.device) * width_scale_q y_q = (image_height - 1) * y0_q + torch.arange(0, resize_height, device=out.device) * height_scale_q

[0120] wherein x_q represents a set of fixed-point cell position information of the x direction of the top-left vertex of each of the N interest units, image_width represents the width of the image to be recognized, x0_q represents the fixed-point region position information of the x direction of the top-left vertex of the interest region, resize_width represents the width of the region feature map, width_scale_q represents the width of the region feature map, y_q represents a set of fixed-point cell position information of the y direction of the top-left vertex of each of the N interest units, image_height represents the height of the image to be recognized, y0_q represents the fixed-point region position information of the y direction of the top-left vertex of the interest region, resize_height represents the height of the region feature map, height_scale_q represents the height scaling coefficient, x0_q * (image_width - 1) represents the position information of the x direction of the top-left vertex of the interest region in the image to be recognized, and y0_q * (image_height - 1) represents the position information of the y direction of the top-left vertex of the interest region in the image to be recognized.

[0121] It can be understood that the above process is to divide each interest region to obtain the floating-point cell position information of each interest unit, and to quantize the floating-point cell position information based on the quantization parameter to obtain the fixed-point cell position information.

[0122] In some embodiments, after the electronic device 100 determines the fixed-point cell position information of the top-left vertex of each of the interest units, the electronic device 100 performs a clipping operation on the fixed-point cell position information of the top-left vertex of each of the interest units, that is, removes the fixed-point cell position information that exceeds the width and height of the image to be recognized from the set of fixed-point cell position information of the x direction of the top-left vertex of each of the N interest units and the set of fixed-point cell position information of the y direction of the top-left vertex of each of the N interest units, respectively, to obtain the effective fixed-point cell position information of the top-left vertex of each of the N interest units as the fixed-point cell position information.

[0123] Exemplarily, the effective fixed-point cell position information of the top-left corner vertex of the N units of interest can be determined by the following code.

[0124] xx_q = torch.clamp(x_q, 0, (image_width - 1) * qmax)

[0125] yy_q = torch.clamp(y_q, 0, (image_height - 1) * qmax)

[0126] wherein xx_q represents a set of effective fixed-point cell position information of the N units of interest in the x direction, x_q represents a set of fixed-point cell position information of the N units of interest in the x direction, image_width represents the width of the image to be recognized, yy_q represents a set of fixed-point cell quantization position information of the N units of interest in the y direction, y_q represents a set of fixed-point cell positions of the N units of interest in the y direction, image_height represents the height of the image to be recognized, and qmax represents the size quantization parameter of the image to be recognized. It can be understood that the size quantization parameter qmax of the image to be recognized is equal to the quantization parameter scale.

[0127] For example, Figure 6d The top-left corner vertex of each unit of interest and the fixed-point cell position information (xx_q, yy_q) of the top-left corner vertex are shown.

[0128] S504: Determine the fixed-point cell position information of each boundary of each unit of interest, and obtain the position feature value of each unit of interest.

[0129] In some embodiments, the electronic device 100 first determines the fixed-point cell position information of each boundary (such as the upper boundary, the lower boundary, the left boundary and the right boundary) of each unit of interest, and determines the fixed-point cell position information of the top-left corner vertex, the bottom-left corner vertex, the top-right corner vertex and the bottom-right corner vertex of each unit of interest according to the fixed-point cell position information of the upper boundary, the lower boundary, the left boundary and the right boundary of each unit of interest. Then, the electronic device 100 obtains the corresponding feature values in the feature map as the position feature values of each unit of interest according to the fixed-point cell position information of the top-left corner vertex, the bottom-left corner vertex, the top-right corner vertex and the bottom-right corner vertex of each unit of interest.

[0130] Exemplarily, the fixed-point cell position information of the upper boundary, the lower boundary, the left boundary and the right boundary of each unit of interest can be determined by the following code.

[0131] top_y_index_q = yy_q >> qvalue

[0132] bottom_y_index_q = (yy_q + qmax) » qvalue

[0133] left_x_index_q = (xx_q + qmax) » qvalue

[0134] right_x_index_q = (xx_q + qmax) » qvalue

[0135] wherein top_y_index_q represents a set of fixed-point cell position information of the upper boundary of each unit of interest, bottom_y_index_q represents a set of fixed-point cell position information of the lower-left boundary of each unit of interest, left_x_index_q represents a set of fixed-point cell position information of the left boundary of each unit of interest, right_x_index represents a set of fixed-point cell position information of the right boundary of each unit of interest, and qmax = 1*scale.

[0136] It can be understood that qvalue corresponds to the number of bits of the fixed-point cell position information yy_q and xx_q, i.e., the exponent of the quantization parameter scale (2 15 ), for example, qvalue can be 15.

[0137] It can be understood that the operation of right shifting qvalue bits "» qvalue" is to match the fixed-point cell position information with the number of bits of the size information of the image to be recognized, i.e., to correspond the top-left corner, the lower-left corner, the top-right corner and the lower-right corner of each unit of interest to the pixel points of the image to be recognized.

[0138] In some embodiments, if the fixed-point cell position information of each boundary is not an integer after the operation of right shifting qvalue bits "» qvalue", i.e., does not correspond to a pixel point on the image to be recognized, an integer operation is performed on the fixed-point cell position information of each boundary so that each vertex corresponds to a pixel point on the image to be recognized. Exemplarily, the integer operation rule can be: performing a floor operation on the quantized position information of the upper boundary and the left boundary, and performing a ceil operation on the quantized position information of the lower boundary and the right boundary.

[0139] For example, Figure 6d The fixed-point cell position information top_y_index_q of the upper boundary and the fixed-point cell position information left_x_index_q of the left boundary of the first unit of interest are shown.

[0140] Exemplarily, the cell position feature values of the top-left corner vertex, the bottom-left corner vertex, the top-right corner vertex and the bottom-right corner vertex of each unit of interest can be determined by the following codes.

[0141] top_left = feature[0, top_y_index_q[idxh], left_x_index_q[idxw], :] # Q12

[0142] top_right = feature[0, top_y_index_q[idxh], right_x_index_q[idxw], :] # Q22

[0143] bottom_left = feature[0, bottom_y_index_q[idxh], left_x_index_q[idxw], :] # Q11

[0144] bottom_right = feature[0, bottom_y_index_q[idxh], right_x_index_q[idxw], :] # Q21

[0145] wherein, top_left represents the position feature value of the top-left corner vertex (Q 12 ) of the unit of interest, top_right represents the position feature value of the top-right corner vertex (Q 22 ) of the unit of interest, bottom_left represents the position feature value of the bottom-left corner vertex (Q 11 ) of the unit of interest, bottom_right represents the position feature value of the bottom-right corner vertex (Q 21 ) of the unit of interest, idxw is greater than 0 and less than resize_width, and idxh is greater than 0 and less than resize_height.

[0146] S505: Determine the cell feature value of each unit of interest based on the fixed-point cell position information and the position feature value of each unit of interest.

[0147] In some embodiments, the electronic device 100 first determines the interpolation coefficient of each unit of interest based on the fixed-point cell position information of the top-left corner vertex, the bottom-left corner vertex, the top-right corner vertex and the bottom-right corner vertex of each unit of interest. Wherein, the interpolation coefficient includes x-direction interpolation coefficient and y-direction interpolation coefficient. Then, the electronic device 100 determines the cell feature value of each unit of interest based on the interpolation coefficient of each unit of interest and the position feature value of the top-left corner vertex, the bottom-left corner vertex, the top-right corner vertex and the bottom-right corner vertex of each unit of interest. The determination of the cell feature value of each unit of interest can refer to the above formula (1).

[0148] Exemplarily, the interpolation coefficient of each unit of interest can be determined by the following code.

[0149] x_lerp_q = (xx_q & 0x7fff).reshape(resize_width, 1).repeat(1, channel)

[0150] y_lerp_q = (yy_q & 0x7fff).reshape(resize_height, 1).repeat(1, channel)

[0151] wherein x_lerp_q represents the x direction interpolation coefficient, and y_lerp_q represents the y direction interpolation coefficient.

[0152] It can be understood that, that is, x lerp represents the distance from the target point (the point corresponding to the unit feature value) to the upper boundary, and y lerp represents the distance from the target point to the left boundary, for example, Figure 6e The x direction interpolation coefficient x_lerp_q and the y direction interpolation coefficient y_lerp_q are shown. Since xx_q and yy_q are floating point quantized fixed point unit position information in the floating point range of 0-1.0, x_lerp_q is the decimal part of xx_q, and y_lerp_q is the decimal part of yy_q. Then x_lerp_q and y_lerp_q are multiplied by the quantization parameter scale to be quantized (i.e. left shift qvalue bits). Then x_lerp_q and y_lerp_q can be determined by performing bitwise AND operation on the fixed point unit position information xx_q and yy_q and the number with qvalue bits being 1, respectively, that is, the last qvalue bits of the fixed point unit position information xx_q and yy_q are determined as x_lerp_q and y_lerp_q, respectively.

[0153] Exemplarily, the unit feature value of the N units of interest can be determined by the following code.

[0154] xy_q = y_lerp_q[idxh, :]*x_lerp_q[idxw, :] >> qvalue

[0155] fourpoint_sum = (top_left + bottom_right - top_right - bottom_left) * xy_q

[0156] top = (top_left << qvalue) + ((top_right - top_left) * x_lerp_q[idxw, :])

[0157] bottom = (bottom_left - top_left) * y_lerp_q[idxh, :]

[0158] data_q = (fourpoint_sum + top + bottom) » qvalue

[0159] resize_feature[boxidx, idxh, idxw, :] = data_q

[0160] wherein, corresponding to the variables in formula 1, xy_q represents x_lerp_q * y_lerp_q; top_left represents Q12; bottom_right represents Q21; top_right represents Q22; bottom_left represents Q11; fourpoint_sum represents (Q12 + Q21 - Q22 - Q11) * x_lerp_q * y_lerp_q; top represents Q12 + (Q22 - Q12) * x_lerp_q; bottom represents (Q11 - Q12) * y_lerp_q; data_q represents f(x, y).

[0161] It can be understood that in “xy_q = y_lerp_q[idxh, :] * x_lerp_q[idxw, :] » qvalue”, the right shift of qvalue bits can effectively prevent overflow in the subsequent calculation process.

[0162] It can be understood that in “top = (top_left << qvalue) + ((top_right - top_left) * x_lerp_q[idxw, :]))”, “top_left << qvalue” is to make the bit number of the data on the left and right sides of the plus sign equal. For example, the bit number of top_left and top_right is A, and the bit number of x_lerp_q is B, then the bit number of the calculation result of the right side of the plus sign ((top_right - top_left) * x_lerp_q[idxw, :])) is A+B, and the bit number of the left side of the plus sign top_left is A, in order to make the bit number of the left and right sides of the plus sign equal, the bit number of top_left should be expanded, that is, left shift B bits. Wherein, qvalue represents the bit number of the fixed point unit position information, the bit number B of x_lerp_q is equal to the bit number of the fixed point region position information of the region of interest.

[0163] It can be understood that "data_q = (fourpoint_sum + top + bottom) >> qvalue" is because the bits of the result need to be restored after being left shifted in the previous calculation.

[0164] S506: Identify the to-be-identified image based on the unit feature values in the region feature map and determine an identification result.

[0165] In some embodiments, the electronic device 100 identifies the to-be-identified image based on the unit feature values in the obtained region feature map and outputs an identification result.

[0166] It can be understood that in some other embodiments, the above steps can be combined, deleted, or replaced by other steps conducive to achieving the purpose of the present application according to actual needs, for example, the above steps S502 and S503 can be combined into one step, which is not limited herein. Figure 5 The steps shown in the above embodiments can be combined, deleted, or replaced by other steps conducive to achieving the purpose of the present application, for example, the above steps S502 and S503 can be combined into one step, which is not limited herein.

[0167] The determination process of formula (1) will be introduced below in combination with the schematic diagram of the bilinear interpolation shown in Figure 4c

[0168] Taking one unit of interest as an example, it can be known from Figure 4c that Q 11 , Q 12 , Q 21 , Q 22 are four vertices of the unit of interest, R1 and R2 represent interpolation points, P represents a target point, x1 represents the position information of Q 11 , Q 12 in the x direction, x represents the position information of R1, P, and R2 in the x direction, and x2 represents the position information of Q 21 , Q 22 in the x direction, y1 represents the position information of Q 11 , Q 21 in the y direction, y represents the position information of P in the y direction, and y2 represents the position information of Q 12 , Q 22 in the y direction.

[0169] First, interpolation in the x direction is performed to obtain two points R1 and R2, the feature value corresponding to R1 can be referred to formula (2), and the feature value corresponding to R2 can be referred to formula (3).

[0170]

[0171]

[0172] wherein R1 represents the feature value corresponding to point R1, f() represents a feature value function, Q 11 , Q​12 , Q 21 , Q 22 are four vertices of the unit of interest, R1, R2 represent interpolation points, x1 represents the position information of R1, R2 in the x direction, x2 represents the position information of Q 11 , Q 12 are four vertices of the unit of interest, x1 represents the position information of Q 21 , Q 22 are four vertices of the unit of interest, y1 represents the position information of Q 11 , Q 21 are four vertices of the unit of interest, y2 represents the position information of Q 12 , Q 22 are four vertices of the unit of interest.

[0173] Then, interpolation in the y direction is performed to obtain the target point P, and the characteristic value corresponding to P can be referred to formula (4).

[0174]

[0175] wherein f() represents a characteristic value function, Q 11 , Q 12 , Q 21 , Q 22 are four vertices of the unit of interest, x1 represents the position information of Q 11 , Q 12 are four vertices of the unit of interest, x represents the position information of the target point P in the x direction, x2 represents the position information of Q 21 , Q 22 are four vertices of the unit of interest, y1 represents the position information of Q 11 , Q 21 are four vertices of the unit of interest, y represents the position information of the target point in the y direction, y2 represents the position information of Q 12 , Q 22 are four vertices of the unit of interest.

[0176] Simplifying the above formula (4), the above formula (1) can be obtained: f(x, y) = Q12*(1-x lerp )*(1-y lerp )+Q22*x lerp *(1-y lerp )+Q11*(1-x lerp )*y lerp +Q21*x lerp *y lerp .

[0177] wherein f(x, y) represents the characteristic value of the target point, Q11 is the characteristic value corresponding to point Q 11 , Q12 is the characteristic value corresponding to point Q 12 , Q21 is the characteristic value corresponding to point Q 21The corresponding eigenvalue, Q22, is point Q. 22 The corresponding eigenvalues, The x-axis interpolation coefficient represents the ratio of the distance between the target point and the nearest feature point in the x-direction to the distance between the feature points above and below it. lerp This represents the ratio of the distance in the y-direction between the target point and the nearest feature point to the distance between the left and right feature points (y-direction interpolation coefficient).

[0178] It is understandable that the bilinear interpolation method can also determine the target point by first interpolating in the y-direction and then interpolating in the x-direction.

[0179] Furthermore, Figure 7 A schematic diagram of the structure of an electronic device 100 is shown according to some embodiments of this application. For example... Figure 7 As shown, the electronic device 100 includes one or more processors 101, system memory 102, non-volatile memory (NVM) 103, communication interface 104, input / output (I / O) devices 105, and system control logic 106 for coupling the processor 101, system memory 102, NVM 103, communication interface 104, and input / output (I / O) devices 105. Wherein:

[0180] The processor 101 can be used to control an electronic device to execute the image processing method of this application, wherein the processor 101 may include a plurality of processing units.

[0181] System memory 102 is volatile memory, such as random-access memory (RAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc. System memory is used for temporary storage of data and / or instructions.

[0182] The non-volatile memory 103 can include one or more tangible, non-transitory computer-readable media for storage of data and / or instructions. In some embodiments, the non-volatile memory 103 can include any suitable non-volatile memory and / or any suitable non-volatile storage device, such as flash memory, and / or a hard disk drive (HDD), compact disc (CD), digital versatile disc (DVD), solid-state drive (SSD), and / or the like. In some embodiments, the non-volatile memory 103 can also be a removable storage medium, such as a secure digital (SD) memory card, and / or the like.

[0183] In particular, the system memory 102 and the non-volatile memory 103 can include, respectively, a temporary copy and a permanent copy of the instructions 107. The instructions 107 can include instructions that, when executed by the processor 101, cause the electronic device 100 to implement the image processing method provided by the embodiments of the present application.

[0184] The communication interface 104 can include a transceiver to provide a wired or wireless communication interface to the electronic device 100 to enable communication to and from any other suitable device via one or more networks. In some embodiments, the communication interface 104 can be integrated with other components of the electronic device 100, such as the communication interface 104 can be integrated with the processor 101. In some embodiments, the electronic device 100 can communicate with other devices via the communication interface 104, such as the electronic device 100 can obtain an image processing method to be executed from other electronic devices via the communication interface 104.

[0185] The input / output (I / O) device 105 can include input devices such as a keyboard, mouse, and / or the like, and / or output devices such as a display, and / or the like, through which a user can interact with the electronic device 100.

[0186] The system control logic 106 can include any suitable interface controllers to provide any suitable interfaces to other modules of the electronic device 100. For example, in some embodiments, the system control logic 106 can include one or more memory controllers to provide an interface to connect to the system memory 102 and the non-volatile memory 103.

[0187] In some embodiments, at least one of the processors 101 can be packaged together with logic for one or more controllers of the system control logic 106 to form a system in package (SiP). In other embodiments, at least one of the processors 101 can also be integrated on the same chip with logic for one or more controllers of the system control logic 106 to form a system on chip (SoC).

[0188] It is to be understood that Figure 7 The structure of the electronic device 100 shown is only one example, and in other embodiments, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangements of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0189] Embodiments of the mechanisms disclosed herein can be implemented in hardware, software, firmware, or a combination of such implementation approaches. Embodiments of the application can be implemented as computer programs or program code executing on programmable systems comprising at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0190] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices, in known fashion. For purposes of this application, a processing system includes any system that has a processor, such as for example a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0191] The program code can be implemented in a high level procedural or object oriented programming language to communicate with a processing system. The program code can be implemented in assembly or machine language, if desired. In fact, the mechanisms described herein are not limited in scope to any particular programming language. In any case, the language can be a compiled or interpreted language.

[0192] In some cases, the disclosed embodiments can be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments can also be implemented as instructions carried by or stored on a transitory or non-transitory machine-readable (e.g., computer-readable) medium, which can be read and executed by one or more processors. For example, the instructions can be distributed over the network or by other computer readable media. Thus, a machine-readable medium can include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including without limitation, floppy diskettes, optical disks, optical fiber, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards, flash memory, or tangible or other machine-readable media. Accordingly, a machine-readable medium includes any medium that is capable of storing or transmitting electronic instructions or information that can be used to program a computer to perform any of the methods disclosed herein.

[0193] In the drawings, some of the structural or methodological features can be shown in particular arrangements and / or orders. However, it should be understood that such particular arrangements and / or orders can not be required. Instead, in some embodiments, the features can be arranged in a different manner and / or order than shown in the illustrative drawings. Additionally, inclusion of structural or methodological features in a particular figure is not meant to imply that such features are required in all embodiments, and in some embodiments, these features can not be included or can be combined with other features.

[0194] It should be noted that each unit / module mentioned in the embodiments of the present application is a logical unit / module, and in the physical world, one logical unit / module can be one physical unit / module, or a part of a physical unit / module, or a combination of multiple physical unit / modules, and the physical implementation of the logical unit / module itself is not the most important. The combination of the functions implemented by these logical units / modules is the key to solving the technical problems proposed in the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned embodiments of the devices of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed in the present application, which does not mean that the above-mentioned device embodiments do not have other units / modules.

[0195] It has to be understood that in the examples and the description of the patent, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusions, so that processes, methods, articles or devices that comprise a list of elements not only include those elements, but also other elements not explicitly listed, or other elements inherent to such processes, methods, articles or devices. Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0196] While the application has been illustrated and described in detail in the drawings and the foregoing description, it should be clear to those of ordinary skill in the art that various changes can be made without departing from the spirit and scope of the application.

Claims

1. An image processing method, characterized in that, include: Obtain the feature map of the image to be identified; Obtain the floating-point region position information of the region of interest on the image to be identified, and quantize the floating-point region position information into fixed-point region position information; Based on the fixed-point location information of the region of interest, the floating-point unit location information of the units of interest included in the region of interest is obtained; The floating-point unit position information is quantized to obtain the fixed-point unit position information; Based on the location information of the fixed-point unit, the feature values ​​of the corresponding unit of interest are obtained; Based on the feature values ​​of the units of interest in the region of interest, a region feature map of the region of interest is obtained; Based on the region feature map, the recognition result of the image to be recognized is obtained.

2. The method according to claim 1, characterized in that, The floating-point region location information includes the position coordinates of multiple vertices of the region of interest.

3. The method according to claim 1, characterized in that, The step of obtaining the floating-point region location information of the region of interest on the image to be identified, and quantizing the floating-point region location information into fixed-point region location information, includes: Obtain the floating-point region position information of the region of interest on the image to be identified; The quantization parameters are determined, and the floating-point region position information is quantized based on the quantization parameters to obtain the fixed-point region position information.

4. The method according to claim 2, characterized in that, The step of obtaining the floating-point unit position information of the units of interest included in the region of interest based on the fixed-point region position information of the region of interest includes: The size information of the region feature map is determined, and the region of interest is divided into multiple units of interest based on the size information of the region feature map and the location information of the fixed point region; Determine the floating-point unit location information for each of the units of interest.

5. The method according to claim 4, characterized in that, The determination of the size information of the region feature map, and the division of the region of interest into multiple units of interest based on the size information of the region feature map and the location information of the fixed-point region, includes: Determine first size information of the region of interest and second size information of the region feature map, wherein the first size information includes the width and height of the region of interest, and the second size information includes the width and height of the region feature map; A height scaling factor is determined based on the height of the region of interest and the height of the region feature map, and a width scaling factor is determined based on the width of the region of interest and the width of the region feature map. The height scaling factor represents the ratio of the height of the region of interest to the height of the region feature map, and the width scaling factor represents the ratio of the width of the region of interest to the width of the region feature map. Based on the height scaling factor and the width scaling factor, the location information of the fixed-point region of interest and the size information of the region feature map, the region of interest is divided into multiple units of interest.

6. The method according to claim 5, characterized in that, Based on the height scaling factor and the width scaling factor, the fixed-point location information of the region of interest, and the size information of the region feature map, the region of interest is divided into multiple units of interest, including: Based on the position coordinates of the vertex at the top left corner of the region of interest, the position of the vertex at the top left corner of the region of interest in the image to be identified is determined; Based on the size information of the region feature map, the height scaling factor, and the width scaling factor, the height and width information of each unit of interest are determined. Based on the position of the top-left vertex of the region of interest in the image to be identified, and the height and width information of each unit of interest, the floating-point unit position information of each unit of interest in the image to be identified is determined, wherein the floating-point unit position information includes the position information of multiple boundaries of the unit of interest.

7. The method according to claim 5, characterized in that, The step of determining a height scaling factor based on the height of the region of interest and the height of the region feature map, and determining a width scaling factor based on the width of the region of interest and the width of the region feature map, includes: The height and width of the region of interest are increased by a preset ratio to obtain a first height and a first width; Determine the divisibility result between the first height and the height of the region feature map to obtain the height divisibility result; Determine the integer division result between the first width and the width of the region feature map to obtain the width division result; The height and width division results are reduced by a preset ratio to obtain the height scaling factor and the width scaling factor.

8. The method according to claim 2, characterized in that, The step of quantizing the floating-point unit position information to obtain fixed-point unit position information includes: The floating-point unit position information is quantized based on the quantization parameters to obtain the fixed-point unit position information.

9. The method according to claim 1, characterized in that, The step of obtaining the feature values ​​of the corresponding unit of interest based on the location information of the fixed-point unit includes: Determine the pixel in the image to be identified corresponding to the location information of the fixed-point unit; The feature values ​​of the pixels in each of the images to be identified are obtained from the feature map and used as the position feature values ​​of the unit of interest. Based on the location information of the fixed-point unit and the location feature value of the unit of interest, the unit feature value of the unit of interest is determined.

10. The method according to claim 9, characterized in that, The step of determining the element feature value of the element of interest based on the fixed-point element location information and the position feature value of the element of interest includes: The first interpolation coefficient and the second interpolation coefficient of each of the units of interest are determined based on the positional feature values ​​of the units of interest; Based on the first interpolation coefficient, the second interpolation coefficient, and the position feature value of the element of interest, the element feature value of the element of interest is determined.

11. The method according to claim 10, characterized in that, The step of determining the first interpolation coefficient and the second interpolation coefficient of each of the units of interest based on their positional feature values ​​includes: Determine the number of bits n of the binary number corresponding to the location information of the fixed-point unit; Wherein, the number of bits in the binary number corresponding to the location information of the fixed point unit is equal to the number of bits in the binary number corresponding to the location information of the fixed point area; The last n bits of the binary number corresponding to the position information of the left boundary of the unit of interest are determined and used as the first interpolation coefficient; The last n bits of the binary number corresponding to the position information of the upper boundary of the unit of interest are determined and used as the second interpolation coefficient.

12. A readable medium, characterized in that, The readable medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the method of any one of claims 1 to 11.

13. An electronic device, characterized in that, include: Memory, used to store instructions executed by one or more processors of an electronic device, and A processor is one of the processors in an electronic device, used to perform the method according to any one of claims 1 to 11.

Images