Image recognition method, medium and electronic device

Abstract

The application relates to the field of image processing, and discloses an image recognition method, a medium and an electronic device, which can filter a large number of invalid processing results in the image recognition process and improve overall computing efficiency. The method comprises the following steps: acquiring an image to be recognized; and recognizing the image to be recognized by using a neural network model to obtain a recognition result. The recognition of the image to be recognized by using the neural network model to obtain the recognition result comprises the following steps: determining a plurality of prediction bounding boxes in the image to be recognized, and calculating a plurality of class scores of each prediction bounding box; selecting target class scores that meet a score condition from the plurality of class scores of the plurality of prediction bounding boxes; and performing normalization processing on the target class scores corresponding to each prediction bounding box. The method can be specifically applied to the scene of class score normalization in image target detection.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image recognition method, medium, and electronic device. Background Technology

[0002] YOLO (a type of neural network) is one of the best performing algorithms in the field of image-based object detection (OD) and has been widely used in artificial intelligence and computer vision.

[0003] The purpose of object detection is to determine the location and category of objects in an input image. The YOLO network can directly generate object category scores (i.e., category probabilities) and location coordinates, and the results can be obtained directly in a single detection.

[0004] In the YOLO network, some layers can calculate a large number of class scores for objects in the input image and input these scores into the region layers. Currently, the region layers normalize the calculated class scores before performing a threshold filtering process. Class scores greater than the threshold (denoted as threshold y1) can proceed to the next step of the YOLO network, while scores less than or equal to the threshold are discarded. Specifically, the normalization process uses the normalized exponential function (softmax) algorithm, which includes addition, division, and exponentiation operations, i.e., it involves many computational steps.

[0005] Therefore, since the number of category scores input to the Region layer is large, and each category score entering the Region layer currently requires various operations such as division in the softmax algorithm, the overall computational load of the Region layer is large, resulting in low data processing efficiency of the Region layer. Summary of the Invention

[0006] This application provides an image recognition method, medium, and electronic device that can filter out a large number of invalid processing results during the image recognition process and improve overall computational efficiency.

[0007] In a first aspect, embodiments of this application provide an image recognition method, which includes: acquiring an image to be recognized; using a neural network model to recognize the image to be recognized, and obtaining a recognition result; wherein, using a neural network model to recognize the image to be recognized and obtaining a recognition result includes: determining multiple predicted bounding boxes in the image to be recognized, and calculating multiple category scores for each predicted bounding box; selecting target category scores that meet the score conditions from the multiple category scores of the multiple predicted bounding boxes; and normalizing the target category scores corresponding to each predicted bounding box. For example, the above normalization process may include the division operation related to the softmax algorithm. It can be understood that the image to be recognized is the input image in the neural network model, and the above is the category represented by the category scores output for objects in the image to be recognized, for example, the recognition result may be the result of an image classification task. In the image recognition process, this application only needs to perform normalization operations, such as division operations, on the selected category scores that meet the score conditions. Thus, it can avoid a large number of invalid operations, such as performing complete normalization operations on the other category scores that do not meet the score conditions. This greatly reduces the number of operations such as division in the normalization process, thus improving image processing efficiency.

[0008] In one possible implementation of the first aspect, the normalization process for the target category score corresponding to each predicted bounding box includes: normalizing the target category score corresponding to each predicted bounding box based on the target category score and the corresponding first value; wherein the first value corresponding to the predicted bounding box is the sum of multiple category scores corresponding to the predicted bounding box. Specifically, after the neural network model calculates the multiple category scores corresponding to each predicted bounding box, for the multiple category scores corresponding to a predicted bounding box, since the division operation in the normalization process (such as the softmax algorithm) is based on the result of the summation operation, to ensure the validity of the calculated category scores, these category scores can be summed first.

[0009] In one possible implementation of the first aspect, the normalization process for the target class score corresponding to each predicted bounding box, based on the target class score and the corresponding first value, includes: performing a division operation between the exponent of the target class score corresponding to each predicted bounding box and the exponent of the corresponding first value to obtain the result of the normalization process for the target class score corresponding to each predicted bounding box. That is, the above normalization process can be the softmax algorithm.

[0010] In one possible implementation of the first aspect, the method further includes setting the category scores that do not meet the score criteria to 0 or deleting them from the multiple category scores of multiple predicted bounding boxes. This avoids normalizing these category scores and reduces computational load.

[0011] In one possible implementation of the first aspect, selecting a target category score that satisfies a score condition from multiple category scores of multiple predicted bounding boxes includes: for each predicted bounding box, selecting a target category score greater than a corresponding first threshold. The first threshold for each predicted bounding box is obtained by multiplying a corresponding first value by a second threshold, where the second threshold is a pre-set threshold in the neural network model. The first value for each predicted box is obtained by summing the corresponding multiple category scores. For example, the first threshold can be threshold y2 (described below), and the second threshold can be threshold y1 (described below), ensuring that the actual physical meaning of the threshold remains unchanged after shifting the normalization operations (such as division).

[0012] In one possible implementation of the first aspect, the multiple category scores for each predicted bounding box are fixed-point numbers; the first threshold corresponding to each predicted bounding box is obtained by multiplying the corresponding first value by the second threshold and dividing by the third value; based on the target category score and the corresponding first value for each predicted bounding box, the target category score for each predicted bounding box is normalized, including: normalizing the target category score for each predicted bounding box based on the target category score, the corresponding first value, and the third value, wherein the third value is related to the data format of the fixed-point number. In this case, the above neural network model can be applied to a fixed-point platform. For example, the third value can be a preset value related to the fixed-point number, such as 255. Thus, the normalized category scores are quantized, so that the processed fixed-point number category scores can be converted to floating-point number form.

[0013] In one possible implementation of the first aspect, the first threshold corresponding to each predicted bounding box is obtained by right-shifting the result of multiplying the corresponding first value by the second threshold by several bits, and the values ​​of the several bits correspond to the third value. For example, the several bits can be 8 bits.

[0014] In one possible implementation of the first aspect, the multiple category scores for each predicted bounding box are all floating-point numbers. That is, the neural network model described above can be applied to a floating-point platform.

[0015] In one possible implementation of the first aspect, the aforementioned neural network model is a YOLO network, and the recognition result of the image to be recognized is related to the target detection result of the image to be recognized. In this case, the above-mentioned selection of category scores that meet the score conditions, and the subsequent normalization process, can be handled by the Region layer in the YOLO network.

[0016] In one possible implementation of the first aspect, determining multiple predicted bounding boxes in the image to be recognized and calculating multiple category scores for each predicted bounding box includes: dividing the image to be recognized into multiple grids; determining multiple predicted bounding boxes corresponding to the multiple grids; and calculating multiple category scores for each predicted bounding box. In this case, the category scores calculated for the multiple predicted bounding boxes corresponding to the multiple grids are relatively large. However, by filtering based on the aforementioned score conditions, a large number of invalid category scores can be filtered out, thereby avoiding invalid normalization processing of these category scores.

[0017] Secondly, embodiments of this application provide a readable medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the image recognition method as described in the first aspect and any possible implementation thereof.

[0018] Thirdly, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, one of the processors of the electronic device, for executing the image recognition method as described in the first aspect and any possible implementation thereof. Attached Figure Description

[0019] Figure 1 According to some embodiments of this application, a schematic diagram of image segmentation under a YOLO network is shown;

[0020] Figure 2 According to some embodiments of this application, a process for filtering data for multiple category scores of a predicted bounding box at a Region layer is shown.

[0021] Figure 3 According to some embodiments of this application, a flowchart of an image recognition method is shown;

[0022] Figure 4 According to some embodiments of this application, a process for filtering data for multiple category scores of a predicted bounding box at a Region layer is shown.

[0023] Figure 5 According to some embodiments of this application, a structural schematic diagram of a mobile phone is shown;

[0024] Figure 6A block diagram of a system is shown according to some embodiments of this application. Detailed Implementation

[0025] The illustrative embodiments of this application include, but are not limited to, image recognition methods, media, and electronic devices.

[0026] In some embodiments, the image recognition method of this application can be applied to image data processing processes that require filtering out a large number of invalid processing results, but is not limited thereto. For example, it can also be applied to audio data processing processes that require filtering out a large number of invalid processing results.

[0027] Specifically, in some embodiments, the above-described image recognition method can be applied to the process of filtering target category scores in image target detection scenarios to filter out invalid category scores. However, the application scenarios of this method are not limited to image target detection scenarios; for example, it can also be used in standalone image classification scenarios.

[0028] In some embodiments, the image recognition method described above can be applied to the process of performing image target detection using a neural network. As an example, the neural network can be a YOLO network, but is not limited thereto.

[0029] Specifically, for object detection tasks in images, some layers in the YOLO network can detect the predicted bounding boxes of objects, detect the class scores (or class probabilities) of each predicted bounding box, and input the class scores and other information of the predicted bounding boxes into the region layer of the YOLO network. Furthermore, the region layer in the YOLO network uses a normalized exponential function (softmax) to normalize and threshold the class scores corresponding to each predicted bounding box. For example, the softmax calculation formula can be found in the following formula (1):

[0030] (1)

[0031] In formula (1) , ,and, , Indicates the first Types, Indicates the first Types, Indicates the total number of categories. This indicates that an object in a predicted bounding box is at the 1st... Scores under each category This indicates that an object in a predicted bounding box is at the 1st... Scores under each category. and This is calculated for the network layers preceding the Region layer in the YOLO network. As can be seen from the softmax formula, the softmax algorithm involves summation, division, and exponentiation operations.

[0032] As can be seen from the background section above, in existing technologies, the Region layer first normalizes the scores of each input category before performing threshold filtering. For example, based on formula (1), for the scores of each category corresponding to a preset bounding box... The processing procedure includes: Step 1: Calculation That is, the scores for each category. index results Perform a summation operation to obtain the summation result; Step 2: For the first... Index results of category scores and the summation result Perform division to calculate the first division number. Normalized scores by category Step 3: Determine the first... Normalized scores by category The system determines whether the score exceeds a threshold (i.e., threshold y1), allowing data with scores exceeding this threshold to proceed to the next step in the YOLO network. This process involves exponential and summation operations followed by division, and finally threshold filtering. However, because the number of category scores input to the Region layer is large, and the background technique requires division for each category score, the overall computational load of the Region layer is significant.

[0033] However, in the aforementioned techniques, some normalized category scores are discarded when they are less than or equal to the threshold y1, rendering the division operation in the softmax algorithm for these category scores invalid. Furthermore, since the number of category scores input to the Region layer is large, and this number far exceeds the actual number of category scores corresponding to the target object, the existing Region layer processing typically involves a large amount of invalid computation. Therefore, how to filter out the large number of invalid computation results in the softmax algorithm of the Region layer to reduce the overall computational load and improve computational efficiency becomes a problem that needs to be solved.

[0034] To address the aforementioned issues, this application provides an image recognition method applicable to scenarios employing YOLO networks for image object detection. The method shifts the computationally intensive parts of the softmax algorithm in the YOLO network to the later stages, primarily by shifting some division operations to the later stages. Specifically, after the YOLO network calculates multiple category scores corresponding to each predicted bounding box, for each predicted bounding box, since the division operation in the softmax formula is based on the summation operation, to ensure the validity of the calculated category scores, these category scores can be summed first. Then, it is directly determined whether these category scores are greater than a pre-set first threshold. When a category score is greater than the first threshold, it indicates that the accuracy of that category score in detecting the object category is high, meaning that the category score can be used for the next step of image object detection in the YOLO network, and the softmax algorithm-related division operation can be performed on that category score. Correspondingly, when a category score is less than or equal to the first threshold, it indicates that the accuracy of detecting the object category for that category is low. This category score is redundant, meaning it doesn't need to proceed to the next step of image object detection in the YOLO network. Dividing this category score at this point would be invalid. Therefore, when a category score is less than or equal to the first threshold, it can be set to zero or deleted without performing the softmax algorithm-related division operations. This way, division operations are only performed on category scores greater than the first threshold, and these operations are moved to after the threshold filtering process, avoiding a large number of invalid division operations. This significantly reduces the number of division operations in the Region layer, improving the overall processing efficiency of the Region layer.

[0035] In some embodiments, the above-described image recognition method can be applied to floating-point processing platforms or fixed-point processing platforms, such as the Compass platform (a fixed-point platform). Furthermore, during data processing on a fixed-point processing platform, data quantization can be performed, allowing the processed data to be converted from fixed-point numbers to corresponding floating-point numbers.

[0036] When YOLO networks are applied to fixed-point arithmetic platforms, the low efficiency of division operations on fixed-point platforms leads to low overall processing efficiency of the Region layer due to the large number of division operations. However, the image recognition method provided in this application reduces the number of division operations in the Region layer by moving the computationally intensive division operations in the Region layer to after threshold filtering, thus significantly improving the overall processing efficiency of the Region layer.

[0037] Reference Figure 1The image shown is an example of image segmentation using the YOLO network. Next, refer to... Figure 1 The scheme for image object detection using the YOLO network is explained.

[0038] like Figure 1 As shown, the YOLO network can segment the input image A1 horizontally along the w dimension and vertically along the h dimension to obtain P×Q grids, where each grid has a width of w and a height of h. For example, Figure 1 The example shown shows P and Q as both 7, but it is not limited to this; for example, P and Q could both be 38. Furthermore, the YOLO network can predict B bounding boxes for each grid cell and detect the location coordinates (x, y, w, h), confidence score, and class scores (e.g., K species scores) for each bounding box. For example, Figure 1 The value of B shown is 7, but it is not limited to this; B can also be 3 or other values. The coordinates (x, y) represent the center coordinates of the predicted bounding box, which can be normalized to between 0 and 1 using the grid's side length. The coordinates w and h represent the width and height of the predicted bounding box, which can be normalized to between 0 and 1 using the width and height of the input image, respectively.

[0039] In some embodiments, if the center of an object falls into a grid, that grid is responsible for detecting the object. For example, if the center of object 11 in input image A1 falls into grid 12, the YOLO network predicts two predicted bounding boxes, predicted bounding box 13 and predicted bounding box 14, for that grid.

[0040] In some embodiments, the core network layer in the YOLO network can perform the above-described process of predicting B predicted bounding boxes for each grid cell, and output this information to the Region layer of the YOLO network. Then, the Region layer can normalize the class probabilities of the target objects calculated for each grid cell using the softmax algorithm.

[0041] For example, combining Figure 1 The grid division shown is for reference. Figure 2 The diagram illustrates the process in existing technologies where the Region layer filters data based on multiple class scores for a predicted bounding box. Figure 2 As shown, for Figure 1The predicted bounding box 13 containing object 11 is shown. In existing technologies, the scores of the first to K categories are first subjected to softmax operations such as division to obtain normalized scores for the first to K categories. Then, the normalized scores for the first to K categories are filtered out using a threshold y1 to remove some category scores, and the remaining category scores are fed into the next step of the YOLO network. Figure 2 The addition operation is omitted in the text, but in practice, addition precedes division. This is understandable, based on... Figure 2 The method described above performs division operations on the 1st to Kth category scores corresponding to the predicted bounding box 13, which is computationally intensive. Furthermore, some division operations in the normalized 1st to Kth category scores are invalid. Therefore, to address these invalid operations, the method described below is used. Figure 3 The image recognition method shown performs filtering, thereby reducing the overall computational load of the Region layer.

[0042] Reference Figure 3 The diagram shown is a flowchart illustrating an image recognition method provided in an embodiment of the application. Next, in conjunction with... Figure 1 and Figure 3 The image recognition method provided in the embodiments of this application will be described. This method can be applied to image target detection scenarios using YOLO networks. Furthermore, the subject executing this method can be any electronic device suitable for the embodiments of this application.

[0043] Specifically, Figure 3 The method shown includes the following steps:

[0044] S301: Divide the input image into P×Q grids.

[0045] As an example, the input image above can be Figure 1 The input image A1 shown is a data structure that can be divided horizontally along the w dimension and vertically along the h dimension, with each grid having a width of w and a height of h.

[0046] In some embodiments, the input data format (shape) of the YOLO network can be [1, 38, 38, 3, 85]. Here, 1 indicates that the batch processing consists of one image. 38, 38 indicates that the input image is segmented horizontally and vertically along a 38-dimensional axis (i.e., h and w are both 38). 3 indicates the number B of predicted bounding boxes that can be detected per grid cell. 85 indicates the length of the matrix storing the input image. For example, the first 5 bits of the matrix can be used to store the location information and confidence score of the predicted bounding boxes, and the last 80 bits can be used to store the class scores of the predicted bounding boxes. In other words, the information of the predicted bounding boxes can be cached in the form of a matrix.

[0047] S302: For the k-th grid among the above multiple grids, predict B predicted bounding boxes, and determine the K category scores corresponding to the B predicted bounding boxes respectively.

[0048] In some embodiments, this application may also determine the location coordinates (x, y, w, h) and confidence of each predicted bounding box. As an example, for an object in a predicted bounding box, the confidence of each class score of the object can be obtained by multiplying the corresponding class scores by the confidence, and used in subsequent operations of image object detection in the YOLO network.

[0049] As an example, the above multiple grids can be Figure 1 The P×Q grids shown can be used by S302 to select relevant information of the preset bounding box for the grid detection in a sequential order from left to right and top to bottom. This information includes K category scores, location coordinates, and confidence levels. Here, k is assigned the value 1-P×Q.

[0050] S303: Perform a summation operation on the K category scores corresponding to the b-th predicted bounding box of the k-th grid to obtain the first value.

[0051] Here, b can be assigned values ​​from 0 to B in sequence.

[0052] In some embodiments, the summation operation related to the softmax algorithm cannot be moved to the later part in order to maintain the validity of the data. It can be understood that the summation operation of the K category scores to obtain the first value can be equivalent to the addition operation in the softmax formula (1).

[0053] S304: For the i-th category score corresponding to the b-th predicted bounding box of the k-th grid, determine whether the category score is greater than the preset threshold y2.

[0054] If the score of the i-th category is determined to be greater than the preset threshold y2, then proceed to S305, where division and exponentiation operations related to the softmax algorithm are performed. Conversely, if the score of the i-th category is determined to be less than or equal to the preset threshold y2, then proceed to S306 without performing division and exponentiation operations related to the softmax algorithm.

[0055] S305: Perform an exponential operation on the i-th category score corresponding to the b-th predicted bounding box of the k-th grid to obtain a second value, perform an exponential operation on the first value to obtain a third value, and perform a division operation on the second and third values ​​to obtain the normalized i-th category score.

[0056] Here, i can be assigned values ​​from 1 to K in sequence.

[0057] It can be understood that the normalized i-th category score can be equivalent to the addition operation in the softmax formula (1) above. Correspondingly, the second and third values ​​mentioned above are respectively equivalent to the values ​​in formula (1). and At this point, S305 performs the exponentiation and division operations related to the softmax algorithm.

[0058] Compared to the solutions in the background technology, S305 moves the division and exponentiation operations in the softmax operation that are greater than the preset threshold (i.e., threshold y2) to after the threshold filtering process, which greatly reduces the number of division and exponentiation operations and thus improves the processing efficiency of the Region layer.

[0059] In some embodiments, after the division operation in the softmax algorithm is shifted, the corresponding threshold value also needs to be transformed to ensure that the actual physical meaning of the threshold remains unchanged. As an example, the threshold y2 mentioned above can be calculated based on the threshold y1 mentioned above and the first value mentioned above, such as multiplying the threshold y1 by the first value to obtain the threshold y2.

[0060] In some embodiments, in a fixed-point arithmetic platform, before filtering data by threshold y1 as described in the background section above, the data is multiplied by Y (e.g., 255) during division. At this point, threshold y2 can be obtained by multiplying the first value by threshold y1 and then dividing by Y. For example, threshold y2 can be obtained by right-shifting the result of multiplying the first value by threshold y1 by N (e.g., 8). Here, Y and N are related to fixed-point numbers.

[0061] S306: Set the i-th category score corresponding to the b-th predicted bounding box of the k-th grid to 0 or delete it.

[0062] It is understandable that if the score of the i-th category is set to 0 or deleted, it will no longer participate in the division operation of the softmax algorithm, and may also not participate in the related exponentiation operation.

[0063] In some embodiments, after some class scores corresponding to a preset bounding box are set to 0, the other class scores corresponding to the predicted bounding box remain valid, and the predicted bounding box will proceed to the next step in the YOLO network. In other embodiments, after all class scores corresponding to a preset bounding box are set to 0, the predicted bounding box will also be invalidated. In this case, the location coordinates and confidence scores corresponding to the predicted bounding box can also be deleted, and thus the predicted bounding box will not proceed to the next step in the YOLO network.

[0064] For example, combining Figure 1 The grid division shown is for reference. Figure 4 As shown, the Region layer in this embodiment performs data filtering on multiple category scores for a predicted bounding box. Figure 4 As shown, for Figure 1 The predicted bounding box 13 showing the object 11 can be processed by first filtering the corresponding scores of the first to K categories using a threshold y2 to remove some category scores. The remaining category scores are then fed into softmax operations such as division to obtain normalized remaining category scores, which are then fed into the next step of the YOLO network. Figure 4 The addition operation is omitted in the original text; in practice, an addition operation occurs before the filtering process using the threshold y2. Clearly, compared to... Figure 2 compared to, Figure 4 The demonstrated process reduces the number of division operations, improving the computational efficiency of the Region layer, especially significantly improving the computational efficiency of the Region layer on a fixed-point arithmetic platform.

[0065] The image recognition method provided in this application addresses the redundancy in the pre-selection box information processed by the Region layer in the YOLO network, and the ineffectiveness of a large number of computationally intensive softmax scores. This is achieved by shifting the computationally intensive division operations in the softmax algorithm to the later stages, specifically after the threshold filtering process. This threshold filtering process filters out a portion of the category scores, significantly reducing the number of category scores requiring division and thus greatly reducing the number of division operations. Consequently, this reduces the computational load of the Region layer, improving computational efficiency, particularly on fixed-point platforms.

[0066] The image recognition method of this application will be explained next in conjunction with the specific implementation of the Region layer.

[0067] It is understandable that in practical applications, the softmax algorithm and threshold filtering in the Region layer can be implemented and executed in the form of software code.

[0068] As an example, in Figure 1 Based on the shown mesh division, the pseudocode (denoted as pseudocode M1) for the softmax algorithm related operations of the Region layer in the background technology above is as follows:

[0069] for row <- 0 to the number of grid cells Q in the h-dimensional segmentation of the image:

[0070] for col <- 0 to the number of grids P that the image is segmented in dimension w;

[0071] for box <- 0 to the number of bounding boxes predicted for each grid cell, B;

[0072] score = sum(score_i)

[0073] score_i = exp(score_i) / exp(score)

[0074] if score_i <= threshold

[0075] score_i=0

[0076] Specifically, the pseudocode M1 can perform softmax algorithm-related operations on multiple grids that divide the input image horizontally along dimension w and vertically along dimension h, including addition, division, and exponentiation. Furthermore, pseudocode M1 is used to process multiple class scores corresponding to the b-th predicted bounding box in the row-th row and column-th column of the P×Q grids of the input image using the softmax algorithm and threshold filtering.

[0077] Next, we will explain each line of code in pseudocode M1:

[0078] First, lines 1-3 of the code in M1 are used to predict the b-th bounding box corresponding to the grid in the row-th row and column-th column of the P×Q grid of the input image.

[0079] In this context, `row` is assigned values ​​from 0 to Q, representing the horizontal index (x-coordinate) of the current grid within a P×Q grid, where Q is the number of grids in the input image segmented along the h-axis. `col` is assigned values ​​from 0 to P, representing the vertical index (y-coordinate) of the current grid within a P×Q grid, where P is the number of grids in the input image segmented along the w-axis. `b` is assigned values ​​from 0 to B, representing the index of the current predicted bounding box within the multiple predicted bounding boxes corresponding to the current grid, for example, B being 2 or 3.

[0080] Secondly, line 4 of the code in M1 is used to sum the scores of each category corresponding to the current b-th predicted bounding box to obtain the total score.

[0081] Here, "score_i" represents the score of the i-th class corresponding to the b-th predicted bounding box, that is, the score obtained by the input data for each grid cell under each class. For example, score_i in the 4th row is equivalent to the softmax formula (1) above. .

[0082] Then, in line 5 of M1, "exp(score_i)" is used to perform an exponential operation on the current i-th category score, "exp(score)" is used to perform an exponential operation on the sum of the above scores, and exp(score_i) / exp(score) is used to perform a division operation on the two exponential operation results to complete the normalization process of the i-th category score. Here, exp() is used to represent an exponential operation with e as the minimum. For example, exp(score_i) / exp(score) in line 5 is equivalent to the softmax formula (1) above. And score_i is equivalent to the value in formula (1). .

[0083] Finally, lines 6-7 of M1 determine whether the current normalized score of the i-th class, score_i, is less than or equal to the threshold y1. If so, the score_i is set to 0. It can be understood that if the score_i is greater than the threshold y1, subsequent operations for image object detection in the YOLO network will proceed.

[0084] It's understandable that line 5 of the pseudocode M1 performs division on all class scores corresponding to the current b-th predicted bounding box. However, most of the normalized class scores will be filtered out by the threshold y1 in lines 6-7 of M1, making the division operation redundant. This results in a large overall computational load and low efficiency for the YOLO network's Region layer.

[0085] In some embodiments, Figure 1 Based on the illustrated mesh division, the pseudocode (denoted as pseudocode M2) for the softmax algorithm-related operations of the Region layer provided in this embodiment of the application is as follows:

[0086] for row <- 0 to the number of grid cells Q in the h-dimensional segmentation of the image:

[0087] for col <- 0 to the number of grids P that the image is segmented in dimension w;

[0088] for box <- 0 to the number of bounding boxes predicted for each grid cell, B;

[0089] score = sum(score_i)

[0090] if score_i > threshold * score

[0091] score_i = exp(score_i) / exp(score)

[0092] The difference between pseudocode M2 ​​and pseudocode M1 is that lines 5-7 of M1 are replaced with lines 5-6 of M2. Lines 1-4 of M1 are identical to lines 1-4 of M2, and will not be repeated here.

[0093] Line 6 of code in M1 determines whether the current normalized score of the i-th category, score_i, is less than or equal to the threshold y2 (i.e., threshold * score). If it is, the current score of the i-th category, score_i, is fed into the subsequent line 7 for exponentiation and division. Otherwise, the current score of the i-th category, score_i, is invalid and does not require division or exponentiation. It can be understood that the i-th category score, score_i, processed by the threshold in line 6 is the unnormalized category score.

[0094] In some embodiments, after the division operation in the softmax algorithm is shifted, the corresponding threshold value also needs to be transformed. For example, the threshold "threshold * score" (i.e., threshold y2) in M2 is obtained by multiplying the threshold "threshold" (i.e., threshold y1) in M1 by the sum of scores (i.e., the first value). Furthermore, for example, in a fixed-point number computation platform, before filtering data by threshold y1, the division operation needs to be multiplied by a fixed-point number-related preset value (such as 255). In this case, the threshold y2 in M2 can be obtained as threshold * score / 255. For example, in the actual computation process, the threshold y2 can be the result of threshold * score shifted right by 8 bits, equivalent to performing threshold * score / 256. Of course, the aforementioned fixed-point number-related preset value is not limited to 255 and can be other values, depending on the actual fixed-point number platform. The purpose of obtaining the threshold y2 based on this preset value in this application is to align the category scores before filtering by threshold y2 with the category scores after the division operation.

[0095] In line 6 of M2, "exp(score_i)" is used to perform an exponential operation on the current i-th category score, "exp(score)" is used to perform an exponential operation on the sum of the above scores, and exp(score_i) / exp(score) is used to perform a division operation on the two exponential operation results to complete the normalization process of the i-th category score. Here, exp() is used to represent the exponential operation with e as the minimum. For example, exp(score_i) / exp(score) in line 5 is equivalent to the softmax formula (1) above. And score_i is equivalent to the value in formula (1). At this point, the normalized score of the i-th class, score_i, can be used for subsequent operations in image object detection within the YOLO network.

[0096] It is understandable that line 5 of the pseudocode M2 ​​above can filter out some class scores through the threshold y1, reducing the number of class scores entering line 6 for division. Furthermore, it also reduces the number of class scores for exponentiation. Therefore, the number of division operations is reduced, resulting in a decrease in the overall computational load of the YOLO network's Region layer and improved efficiency, especially for fixed-point platforms.

[0097] In some implementations, the image recognition method provided in this application can significantly improve the running efficiency of the Region layer on the Compass platform. Taking input data in the format [1,38,38,3,85] (i.e., an input image) as an example, on a 1GHz platform, the original Region layer's running time is 8.86ms, including some unoptimized parts such as data transfer and bounding box correction, approximately 2ms, and the softmax-related parts approximately 6.86ms. After optimization, the Region layer's running time is reduced to 3.452ms, with the optimized part being only 1.5ms, a 22% reduction, resulting in a significant improvement in performance.

[0098] In some embodiments, the image recognition method provided in this application is not limited to optimizing the softmax calculation in the Region. In other data processing scenarios, a large portion of the computational load can still be shifted to the later stages to filter out a large number of invalid results. For example, these data scenarios can be other scenarios where the softmax algorithm is applied, but are not limited to them.

[0099] In some embodiments, the electronic device applicable to the image recognition method provided in this application may be a mobile phone, computer, tablet, camera, smart wearable device, high-definition video live streaming capture device, television or set-top box, wearable device, AR device, VR device, MR device, digital direct broadcasting system, wireless communication device, personal digital assistant (PDA), laptop or desktop computer, video game device, etc., but is not limited thereto. The hardware structure of the electronic device will be described below using a mobile phone as an example.

[0100] like Figure 5 As shown, the mobile phone 10 may include a processor 110, a power module 140, a memory 180, a mobile communication module 130, a wireless communication module 120, a sensor module 190, an audio module 150, a camera 170, an interface module 160, buttons 101, and a display screen 102, etc.

[0101] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the mobile phone 10. In other embodiments of this application, the mobile phone 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0102] Processor 110 may include one or more processing units, such as processing modules or circuits of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), micro-programmed control unit (MCU), artificial intelligence (AI) processor, or field programmable gate array (FPGA). Different processing units may be independent devices or integrated into one or more processors. Processor 110 may include storage units for storing instructions and data. In some embodiments, the storage unit in processor 110 is a cache memory 180. For example, processor 110 may be used to move the division operation in the softmax algorithm of the Region layer in a YOLO network after threshold filtering, thereby reducing the number of operations such as division in the softmax algorithm.

[0103] The power module 140 may include a power supply, a power management component, etc. The power supply may be a battery. The power management component manages the charging of the power supply and the power supply to other modules. In some embodiments, the power management component includes a charging management module and a power management module. The charging management module receives charging input from a charger; the power management module connects to the power supply and the processor 110. The power management module receives input from the power supply and / or the charging management module to supply power to the processor 110, the display 102, the camera 170, and the wireless communication module 120, etc.

[0104] The mobile communication module 130 may include, but is not limited to, antennas, power amplifiers, filters, and LNAs (Low Noise Amplifiers). The mobile communication module 130 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on the mobile phone 10. The mobile communication module 130 can receive electromagnetic waves via the antenna, filter and amplify the received electromagnetic waves, and then transmit them to a modem processor for demodulation. The mobile communication module 130 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna. In some embodiments, at least some functional modules of the mobile communication module 130 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 130 and at least some modules of the processor 110 may be housed in the same device. Wireless communication technologies can include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), Wireless Local Area Networks (WLAN), Near Field Communication (NFC), Frequency Modulation (FM) and / or Field Communication (NFC), Infrared (IR) technology, etc.The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0105] The wireless communication module 120 may include an antenna, which enables the transmission and reception of electromagnetic waves. The wireless communication module 120 can provide solutions for wireless communication applications on the mobile phone 10, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The mobile phone 10 can communicate with networks and other devices through wireless communication technologies.

[0106] In some embodiments, the mobile communication module 130 and the wireless communication module 120 of the mobile phone 10 may also be located in the same module.

[0107] The display screen 102 is used to display human-computer interaction interfaces, images, videos, etc. The display screen 102 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0108] The sensor module 190 may include proximity sensors, pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0109] The audio module 150 is used to convert digital audio information into analog audio signals for output, or to convert analog audio input into digital audio signals. The audio module 150 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 150 may be located in the processor 110, or some functional modules of the audio module 150 may be located in the processor 110. In some embodiments, the audio module 150 may include a speaker, a handset, a microphone, and a headphone jack.

[0110] Camera 170 is used to capture still images or videos. An object passes through the lens to generate an optical image that is projected onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then passed to the ISP (Image Signal Processing) to be converted into a digital image signal. Mobile phone 10 can achieve its shooting function through the ISP, camera 170, video codec, GPU (Graphics Processing Unit), display 102, and application processor.

[0111] Interface module 160 includes an external memory interface, a universal serial bus (USB) interface, and a subscriber identification module (SIM) card interface. The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 10. The external memory card communicates with the processor 110 through the external memory interface to perform data storage. The USB interface is used for communication between the mobile phone 10 and other electronic devices. The SIM card interface is used to communicate with the SIM card installed in the mobile phone 10, for example, to read or write phone numbers stored in the SIM card.

[0112] In some embodiments, the mobile phone 10 further includes buttons 101, a motor, and indicators. The buttons 101 may include volume buttons, a power button, etc. The motor is used to generate a vibration effect in the mobile phone 10, for example, vibrating when the user's mobile phone 10 is called to prompt the user to answer the call. The indicators may include laser indicators, radio frequency indicators, LED indicators, etc.

[0113] Now for reference Figure 6 The diagram shown is a block diagram of a system 1400 according to an embodiment of this application. Figure 6 An example system 1400 according to several embodiments is schematically illustrated. This system 1400 can be applied to an electronic device (such as a mobile phone 10) of this application. In one embodiment, system 1400 may include one or more processors 1404, system control logic 1408 connected to at least one of the processors 1404, system memory 1412 connected to system control logic 1408, non-volatile memory (NVM) 1416 connected to system control logic 1408, and network interface 1420 connected to system control logic 1408.

[0114] In some embodiments, processor 1404 may include one or more single-core or multi-core processors. In some embodiments, processor 1404 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments where system 1400 employs eNB (Evolved Node B) 101 or RAN (Radio Access Network) controller 102, processor 1404 may be configured to perform various conforming embodiments, such as... Figure 3 One or more of the multiple embodiments shown.

[0115] In some embodiments, system control logic 1408 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1404 and / or any suitable device or component communicating with system control logic 1408.

[0116] In some embodiments, system control logic 1408 may include one or more memory controllers to provide an interface to system memory 1412. System memory 1412 may be used to load and store data and / or instructions. In some embodiments, memory 1412 of system 1400 may include any suitable volatile memory, such as suitable dynamic random access memory (DRAM).

[0117] NVM / memory 1416 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM / memory 1416 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of HDD (Hard Disk Drive), CD (Compact Disc) drive, and DVD (Digital Versatile Disc) drive.

[0118] NVM / Storage 1416 may include a portion of the storage resources on the device on which System 1400 is installed, or it may be accessible by the device, but is not necessarily part of the device. For example, NVM / Storage 1416 may be accessed over a network via Network Interface 1420.

[0119] Specifically, system memory 1412 and NVM / memory 1416 may each include a temporary copy and a permanent copy of instruction 1424. Instruction 1424 may include, when executed by at least one of processors 1404, causing system 1400 to perform as follows: Figure 3 The instructions for the method shown. In some embodiments, instructions 1424, hardware, firmware and / or their software components may additionally / alternatively be located in system control logic 1408, network interface 1420 and / or processor 1404.

[0120] Network interface 1420 may include a transceiver for providing a radio interface to system 1400, thereby enabling communication with any other suitable device (such as a front-end module, antenna, etc.) via one or more networks. In some embodiments, network interface 1420 may be integrated into other components of system 1400. For example, network interface 1420 may be integrated into at least one of processor 1404, system memory 1412, NVM / memory 1416, and firmware device (not shown) with instructions that, when at least one of processor 1404 executes the instructions, system 1400 implements as follows: Figure 3 The method shown.

[0121] The network interface 1420 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 1420 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0122] In one embodiment, at least one of the processors 1404 may be packaged together with the logic of one or more controllers for system control logic 1408 to form a system package (SiP). In another embodiment, at least one of the processors 1404 may be integrated on the same die with the logic of one or more controllers for system control logic 1408 to form a system on chip (SoC).

[0123] System 1400 may further include an input / output (I / O) device 1432. The I / O device 1432 may include a user interface enabling a user to interact with system 1400; and the peripheral component interface may be designed to allow peripheral components to also interact with system 1400. In some embodiments, system 1400 may also include sensors for determining at least one type of environmental condition and location information related to system 1400.

[0124] In some embodiments, the user interface may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light-emitting diode flash), and a keyboard.

[0125] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0126] In some embodiments, the sensor may include, but is not limited to, a gyroscope sensor, an accelerometer, a proximity sensor, an ambient light sensor, and a positioning unit. The positioning unit may also be part of or interact with the network interface 1420 to communicate with components of the positioning network, such as Global Positioning System (GPS) satellites.

[0127] Various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or combinations of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0128] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0129] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0130] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0131] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0132] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0133] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. An image recognition method, characterized in that, The method includes: Acquire the image to be recognized; A neural network model is used to identify the image to be identified, and the identification result is obtained; The step of using a neural network model to identify the image to be identified and obtaining the identification result includes: Determine multiple predicted bounding boxes in the image to be identified, and calculate multiple category scores for each predicted bounding box; From the multiple category scores of the multiple predicted bounding boxes, a target category score that meets the score condition is selected; the selection of the target category score that meets the score condition from the multiple category scores of the multiple predicted bounding boxes includes: for each of the multiple category scores of the predicted bounding box, selecting a target category score that is greater than the corresponding first threshold, wherein the first threshold for each predicted bounding box is obtained by multiplying the corresponding first value by a second threshold, the second threshold is a threshold preset in the neural network model, and the first value for each predicted bounding box is obtained by summing the corresponding multiple category scores; The target category score corresponding to each predicted bounding box is normalized. The normalization of the target category score corresponding to each predicted bounding box includes: performing an exponential operation on the target category score corresponding to each predicted bounding box to obtain a corresponding second value, performing an exponential operation on the first value to obtain a corresponding third value, and performing a division operation on the second value and the third value corresponding to each predicted bounding box to obtain the result of normalizing the target category score corresponding to each predicted bounding box.

2. The method according to claim 1, characterized in that, The method further includes: From the multiple category scores of the multiple predicted bounding boxes, the category scores that do not meet the score conditions are set to 0 or deleted.

3. The method according to claim 1, characterized in that, The multiple category scores for each predicted bounding box are fixed-point numbers; The first threshold corresponding to each predicted bounding box is obtained by multiplying the corresponding first value by the second threshold and dividing the result by a preset value related to the number of fixed points. The normalization process for the target category score corresponding to each predicted bounding box includes: An exponential operation is performed on the target category score corresponding to each predicted bounding box to obtain a corresponding second value. An exponential operation is performed on the first value to obtain a corresponding third value. The second value and the third value corresponding to each predicted bounding box are divided and then multiplied by the preset value to obtain the result of normalizing the target category score corresponding to each predicted bounding box. The preset value is related to the data format of the fixed-point number.

4. The method according to claim 3, characterized in that, The first threshold corresponding to each predicted bounding box is obtained by right-shifting the result of multiplying the corresponding first value by the second threshold by multiple bits, and the values ​​of the multiple bits correspond to the preset value.

5. The method according to claim 1, characterized in that, The multiple category scores for each predicted bounding box are floating-point numbers.

6. The method according to claim 1, characterized in that, The neural network model is a YOLO network, and the recognition result of the image to be recognized is related to the target detection result of the image to be recognized.

7. The method according to claim 6, characterized in that, The process of determining multiple predicted bounding boxes in the image to be identified and calculating multiple category scores for each predicted bounding box includes: The image to be identified is divided into multiple grids; Determine multiple predicted bounding boxes corresponding to the multiple grids, and calculate multiple category scores for each predicted bounding box.

8. A readable medium, characterized in that, The readable medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the image recognition method according to any one of claims 1 to 7.

9. An electronic device, characterized in that, include: A memory for storing instructions executed by one or more processors of an electronic device, and a processor, one of the processors of the electronic device, for performing the image recognition method according to any one of claims 1 to 7.

Images