An image recognition method, device, electronic equipment and storage medium

Abstract

This application provides an image recognition method, apparatus, electronic device, and storage medium. The method includes: obtaining a second feature map at at least one scale based on a first feature map corresponding to an image to be recognized; determining layering coefficients corresponding to various scales based on the second feature maps at at least one scale, wherein the layering coefficients are used to characterize the weight of each recognition layer in a recognition module for image recognition of the second feature map at each scale, and the recognition layer is used to perform image recognition on the second feature map at each scale; and selecting a layer to be used from the multiple recognition layers of the recognition module based on the layering coefficients, so as to perform image recognition on the second feature map at each scale based on the layer to be used.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image recognition method, apparatus, electronic device and storage medium. Background Technology

[0002] In image recognition scenarios, the challenge lies in balancing accuracy and computational complexity. Summary of the Invention

[0003] This application provides the following technical solution:

[0004] This application provides an image recognition method, including:

[0005] Based on the first feature map corresponding to the image to be identified, a second feature map of at least one scale is obtained;

[0006] Based on the second feature maps of various scales in the at least one scale, a layering coefficient corresponding to each scale is determined. The layering coefficient is used to characterize the weight of each recognition layer in the multiple recognition layers of the recognition module for image recognition of the second feature map of the scale. The recognition layer is used to perform image recognition on the second feature map of the scale.

[0007] Based on the hierarchical coefficient, a layer to be used is selected from multiple recognition layers of the recognition module to perform image recognition on the second feature map of the scale based on the layer to be used.

[0008] The image recognition method is based on a multi-task autonomous hierarchical model, which includes a feature extraction module, a hierarchical estimation module, and a recognition module.

[0009] The feature extraction module is used to extract the first feature map corresponding to the image to be identified;

[0010] The hierarchical estimation module is used to obtain a second feature map of at least one scale based on a first feature map corresponding to the image to be identified, and to determine the hierarchical coefficients corresponding to each scale based on the second feature maps of each scale among the at least one scale.

[0011] The recognition module is used to select a layer to be used from multiple recognition layers of the recognition module based on the layering coefficient, so as to perform image recognition on the second feature map of the scale based on the layer to be used.

[0012] The hierarchical estimation module includes: a first extraction module and a coefficient determination module;

[0013] The first extraction module is used to extract candidate boxes of at least one scale corresponding to the first feature map of the image to be identified; and,

[0014] Based on the candidate boxes of at least one scale and the first feature map, a second feature map of at least one scale is determined;

[0015] The coefficient determination module is used to determine, based on the second feature maps of various scales in the at least one scale, the first probability that each recognition layer of the recognition module corresponding to each scale can recognize the second feature map of the scale, wherein the first probability represents the weight of the recognition layer for image recognition of the second feature map of the scale.

[0016] The coefficient determination module includes: a second extraction module, a third extraction module, and a classification module;

[0017] Based on the second feature maps of various scales in the at least one scale, determining the first probability that each recognition layer in the multiple recognition layers of the recognition module corresponding to each scale can recognize the second feature map of the scale includes:

[0018] The second feature maps of various scales in the at least one scale are input into the second extraction module to obtain the third feature maps of various scales determined by the second extraction module;

[0019] The third feature maps of various scales are input into the third extraction module to obtain the fourth feature maps of various scales determined by the third extraction module.

[0020] The fourth feature maps of various scales are input into the classification module to obtain the first probability that each of the multiple recognition layers of the recognition module corresponding to the various scales can recognize the second feature map of the scale.

[0021] Based on the hierarchical coefficient, a layer to be used is selected from multiple recognition layers of the recognition module, including:

[0022] Based on the layering coefficient, the recognition layer with the top n weights from the multiple recognition layers of the recognition module is selected as the layer to be used;

[0023] or,

[0024] Based on the layering coefficient, the layer with the highest weight is selected from the multiple recognition layers of the recognition module as the layer to be used.

[0025] The first extraction module, the coefficient determination module, and the recognition module are trained in the following manner:

[0026] Obtain the first sample feature map corresponding to at least one sample image for a task;

[0027] Based on the first extraction module, extract at least one scale of second sample feature map corresponding to the first sample feature map of the sample image;

[0028] The second sample feature maps of various scales in the at least one scale are input into the coefficient determination module to obtain the second probability that each layer of the recognition module corresponding to various scales can recognize the second sample feature map of the scale, as determined by the coefficient determination module. The second probability represents the weight of the layer for image recognition of the second sample feature map of the scale.

[0029] The recognition module is controlled to perform image recognition based on the second probability corresponding to each layer in the recognition module corresponding to the scale and the second sample feature map corresponding to the scale, so as to obtain the image recognition result corresponding to the scale.

[0030] If the loss function value of the recognition module does not converge, the parameters of the first extraction module, the coefficient determination module and the recognition module are adjusted, and the step of obtaining the first sample feature map corresponding to the sample image corresponding to at least one task is returned. The loss function value of the recognition module represents the difference between the image recognition result corresponding to the scale and the real recognition result corresponding to the sample image.

[0031] If the loss function value of the recognition module converges, the training step ends.

[0032] The feature extraction module, the first extraction module, the coefficient determination module, and the recognition module are trained in the following manner:

[0033] Obtain at least one sample image corresponding to a task;

[0034] Based on the feature extraction module, a first sample feature map corresponding to the sample image is extracted;

[0035] Based on the first extraction module, at least one scale of candidate bounding boxes corresponding to the first sample feature map of the sample image are extracted;

[0036] Based on the candidate bounding boxes of the samples at the at least one scale and the first sample feature map, a second sample feature map of at least one scale is determined;

[0037] The second sample feature maps of various scales in the at least one scale are input into the coefficient determination module to obtain the second probability that each layer of the recognition module corresponding to various scales can recognize the second sample feature map of the scale, as determined by the coefficient determination module. The second probability represents the weight of the layer for image recognition of the second sample feature map of the scale.

[0038] The recognition module is controlled to perform image recognition based on the second probability corresponding to each layer in the recognition module corresponding to the scale and the second sample feature map corresponding to the scale, so as to obtain the image recognition result corresponding to the scale.

[0039] If the loss function value of the recognition module does not converge, the parameters of the feature extraction module, the first extraction module, the coefficient determination module and the recognition module are adjusted, and the step of obtaining at least one sample image corresponding to the task is returned. The loss function value of the recognition module represents the difference between the image recognition result corresponding to the scale and the real recognition result corresponding to the sample image.

[0040] If the loss function value of the recognition module converges, the training step ends.

[0041] This application also provides an image recognition device, comprising:

[0042] The obtaining unit is used to obtain a second feature map of at least one scale based on a first feature map corresponding to the image to be identified;

[0043] A determining unit is configured to determine a layering coefficient corresponding to each of the at least one scale based on the second feature maps of each of the various scales. The layering coefficient is used to characterize the weight of each of the multiple recognition layers of the recognition module for image recognition of the second feature maps of the scale. The recognition layers are used for image recognition of the second feature maps of the scale.

[0044] The selection unit is used to select a layer to be used from multiple recognition layers of the recognition module based on the layering coefficient, so as to perform image recognition on the second feature map of the scale based on the layer to be used.

[0045] A third aspect of this application provides an electronic device, including: a memory and a processor;

[0046] The memory is used to store at least one set of instructions;

[0047] The processor is configured to call and execute the instruction set in the memory, and execute the image recognition method as described in any of the above descriptions by executing the instruction set.

[0048] A fourth aspect of this application provides a storage medium storing a computer program that implements the image recognition method as described in any one of the above claims, the computer program being executed by a processor to implement the image recognition method as described in any one of the above claims. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating an image recognition method provided in the first embodiment of this application;

[0051] Figure 2 This is a schematic diagram of an implementation scenario of an image recognition method provided in the first embodiment of this application;

[0052] Figure 3 This is a schematic flowchart of an image recognition method provided in the second embodiment of this application;

[0053] Figure 4 This is a schematic diagram of a training scenario provided in this application;

[0054] Figure 5 This is a schematic diagram illustrating an implementation scenario of an image recognition method provided in the second embodiment of this application;

[0055] Figure 6 This is a flowchart illustrating an image recognition method provided in the third embodiment of this application;

[0056] Figure 7 This is a flowchart illustrating an image recognition method provided in the fourth embodiment of this application;

[0057] Figure 8 This is a flowchart illustrating an image recognition method provided in the fifth embodiment of this application;

[0058] Figure 9 This is a schematic diagram of the structure of an image recognition device provided in this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] Reference Figure 1This is a flowchart illustrating an image recognition method provided in the first embodiment of this application. This method can be applied to electronic devices. This application does not limit the type of electronic device, such as... Figure 1 As shown, the method may include, but is not limited to, the following steps:

[0062] Step S101: Based on the first feature map corresponding to the image to be identified, obtain a second feature map of at least one scale.

[0063] In this embodiment, the first feature map corresponding to the image to be identified can be obtained by feature extraction of the image to be identified.

[0064] The first feature map can represent the image to be identified.

[0065] In this embodiment, feature extraction at at least one scale can be performed on the first feature map corresponding to the image to be recognized to obtain a second feature map at at least one scale.

[0066] Second feature maps at at least one scale can collectively represent the image to be identified.

[0067] Step S102: Based on the second feature maps of various scales in at least one scale, determine the layering coefficients corresponding to various scales. The layering coefficients are used to characterize the weight of each recognition layer in the multiple recognition layers of the recognition module for image recognition of the second feature maps of the scales. The recognition layers are used to perform image recognition on the second feature maps of the scales.

[0068] In this embodiment, the recognition module may include multiple recognition layers, which may be, but are not limited to, the output layer of a deep learning network model. The deep learning network model may also include hidden layers. During the training phase, the output layer and the hidden layers jointly perform image recognition on the second feature map at a specific scale; during the recognition phase, only the output layer is used to perform image recognition on the second feature map at a specific scale.

[0069] For second feature maps of various scales, the weights for image recognition of second feature maps of different scales vary among different recognition layers.

[0070] The greater the weight the recognition layer places on the second feature map at a certain scale for image recognition, the better the recognition layer performs on the second feature map at that scale.

[0071] Step S103: Based on the hierarchical coefficient, select the layer to be used from multiple recognition layers of the recognition module, and perform image recognition on the second feature map of the scale based on the layer to be used.

[0072] In this embodiment, it is possible, but not limited to, selecting one layer to be used from multiple recognition layers of the recognition module based on a hierarchical system.

[0073] Of course, it is also possible, but not limited to, selecting multiple layers to be used from multiple recognition layers of the recognition module based on the layering coefficient.

[0074] It is understandable that if the weights of the second feature map of the layer to be used for image recognition at the scale meet the set weight conditions, the image recognition effect of the second feature map of the layer to be used at the scale is better than the image recognition effect of the second feature map of other recognition layers at the scale.

[0075] In this embodiment, a second feature map at at least one scale is obtained based on a first feature map corresponding to the image to be recognized. Based on the second feature maps at various scales within the at least one scale, layering coefficients corresponding to each scale are determined. Based on these layering coefficients, a layer to be used is selected from multiple recognition layers in the recognition module. This allows for the selection of a layer corresponding to the scale, enabling image recognition based on the second feature map of that scale. This reduces the computational load of image recognition while ensuring accuracy. For example, if the recognition module includes four recognition layers, and the layering coefficients are uncertain, such as... Figure 2 As shown in section (a), the first feature map corresponding to the image to be recognized is input into recognition layer 1, recognition layer 2, recognition layer 3, and recognition layer 4. Recognition layer 1, recognition layer 2, recognition layer 3, and recognition layer 4 perform image recognition on the first feature map corresponding to the image to be recognized to obtain the image recognition result, which involves a large amount of computation. If a second feature map of scale A×A is obtained based on the first feature map corresponding to the image to be recognized, the layering coefficient corresponding to A×A is determined based on the second feature map of A×A. The layering coefficient corresponding to A×A indicates that recognition layer 2 has the largest weight for image recognition on the second feature map of A×A. Figure 2 As shown in section (b), recognition layer 2 is selected as the layer to be used from recognition layer 1, recognition layer 2, recognition layer 3 and recognition layer 4. Image recognition is performed on the second feature map of A×A based on the layer to be used to obtain the image recognition result. It is not necessary to perform image recognition based on recognition layer 1, recognition layer 2, recognition layer 3 and recognition layer 4, which can reduce the amount of computation. Furthermore, using recognition layer 2 with the largest weight for image recognition can ensure the accuracy of image recognition.

[0076] Furthermore, when different tasks focus on different scales, the connection between the task and the recognition layer can be established based on the scale of the second feature map, enabling different recognition layers to perform image recognition corresponding to different tasks. For example, multiple tasks include a first task, a second task, and a third task. The first task is used to recognize smoking behavior, the second task is used to recognize falling behavior, and the third task is used to recognize fighting behavior. The scale of the first task is A×A, the scale of the second task is B×B, and the scale of the third task is C×C. A×A is less than B×B, B×B is less than C×C. In the recognition module, the weight of the first recognition layer for image recognition of the second feature map of A×A is a11, the weight for image recognition of the second feature map of B×B is a12, and the weight for image recognition of the second feature map of C×C is a13. a11 is less than a12, and a12 is less than a13.

[0077] In the recognition module, the weight for image recognition of the second feature map of A×A in the second recognition layer is a21, the weight for image recognition of the second feature map of B×B is a22, and the weight for image recognition of the second feature map of C×C is a23. a21 is less than a23, and a23 is less than a22.

[0078] In the recognition module, the weight for image recognition of the second feature map of A×A in the third recognition layer is a31, the weight for image recognition of the second feature map of B×B is a32, and the weight for image recognition of the second feature map of C×C is a33. a33 is less than a32, and a32 is less than a31.

[0079] The first recognition layer is selected from the first recognition layer, the second recognition layer and the third recognition layer to perform image recognition on the image and identify fighting behavior;

[0080] The image is selected from the first recognition layer, the second recognition layer and the third recognition layer to perform image recognition and identify the fall behavior;

[0081] The third recognition layer is selected from the first, second, and third recognition layers to perform image recognition on the image and identify smoking behavior.

[0082] As another optional embodiment of this application, this embodiment mainly provides a multi-task autonomous hierarchical model. The image recognition method described in the first embodiment above can be executed based on the multi-task autonomous hierarchical model. The multi-task autonomous hierarchical model may include, but is not limited to, a feature extraction module, a hierarchical estimation module, and a recognition module.

[0083] The feature extraction module is used to extract the first feature map corresponding to the image to be identified.

[0084] The hierarchical estimation module is used to obtain a second feature map of at least one scale based on a first feature map corresponding to the image to be identified, and to determine the hierarchical coefficients corresponding to various scales based on the second feature maps of various scales in the at least one scale.

[0085] In this embodiment, the hierarchical estimation module may include, but is not limited to, a first extraction module and a coefficient determination module.

[0086] The first extraction module is used to extract candidate boxes of at least one scale corresponding to the first feature map of the image to be identified; and,

[0087] Based on candidate boxes of at least one scale and a first feature map, a second feature map of at least one scale is determined.

[0088] The coefficient determination module is used to determine, based on the second feature maps of various scales in at least one scale, the first probability that each recognition layer of the recognition module corresponding to each scale can recognize the second feature map of the scale. The first probability represents the weight of the recognition layer for image recognition of the second feature map of the scale.

[0089] The recognition module is used to select a layer to be used from multiple recognition layers of the recognition module based on the hierarchical coefficient, so as to perform image recognition on the second feature map of the scale based on the layer to be used.

[0090] like Figure 3 The diagram shown is a flowchart of an image recognition method provided in the second embodiment of this application. The corresponding hierarchical estimation module includes an implementation of a first extraction module and a coefficient determination module. This embodiment mainly refines steps S101, S102, and S103 of the first embodiment described above. Specifically, step S101 may include, but is not limited to, the following steps:

[0091] Step S1011: Extract candidate boxes of at least one scale corresponding to the first feature map of the image to be identified.

[0092] Candidate boxes can be used to extract features from the first feature map.

[0093] In this embodiment, candidate boxes of at least one scale corresponding to the first feature map of the image to be identified can be extracted based on, but not limited to, the RPN (Region Proposal Network).

[0094] Step S1012: Based on candidate boxes of at least one scale and the first feature map, determine a second feature map of at least one scale.

[0095] In this embodiment, features can be extracted from the first feature map based on the candidate boxes of each scale in the candidate boxes of at least one scale to obtain the second feature map of each scale, thus obtaining the second feature map of at least one scale.

[0096] Accordingly, step S102 in the first embodiment described above may include, but is not limited to, the following steps:

[0097] Step S1021: Based on the second feature maps of various scales in at least one scale, determine the first probability that each recognition layer in the multiple recognition layers of the recognition module corresponding to various scales can recognize the second feature map of the scale. The first probability represents the weight of the recognition layer for image recognition of the second feature map of the scale.

[0098] In this embodiment, the first probability of each recognition layer in the multiple recognition layers of the recognition module recognizing the second feature map of the scale is different from each other.

[0099] Among them, the sum of the first probabilities of each recognition layer in the recognition module being able to recognize the second feature map of the scale is 1.

[0100] The higher the probability that the recognition layer can identify the second feature map of the scale, the better the image recognition effect of the recognition layer on the second feature map of the scale.

[0101] Accordingly, step S103 in the first embodiment described above may include, but is not limited to:

[0102] Step S1031: Based on the first probability that each recognition layer can recognize the second feature map of the scale, select the layer to be used from multiple recognition layers of the recognition module, so as to perform image recognition on the second feature map of the scale based on the layer to be used.

[0103] Step S1031 may include, but is not limited to:

[0104] S10311. Select the recognition layer with the highest probability among the top n recognition layers from the multiple recognition layers of the recognition module as the layer to be used.

[0105] Step S1031 may also include, but is not limited to:

[0106] S10312. Select the recognition layer with the highest probability from the multiple recognition layers of the recognition module as the layer to be used.

[0107] In this embodiment, the first extraction module, coefficient determination module, and recognition module described above can be trained, but are not limited to, in the following ways:

[0108] S11. Obtain the first sample feature map corresponding to at least one sample image of a task.

[0109] In this embodiment, the image recognition corresponding to each task in at least one task is different from each other.

[0110] In this application, at least one task can be configured as needed. For example, at least one task may include a first task for identifying smoking behavior, a second task for identifying falling behavior, and a third task for identifying fighting behavior.

[0111] The first sample feature map can represent the sample image corresponding to the task.

[0112] The first sample feature map can be extracted based on a pre-trained feature extraction module.

[0113] S12. Extract a second sample feature map of at least one scale corresponding to the first sample feature map corresponding to the sample image based on the first extraction module.

[0114] A second sample feature map at at least one scale can jointly represent a sample image.

[0115] S13. Input the second sample feature maps of various scales in at least one scale into the coefficient determination module to obtain the second probability of each layer of the recognition module corresponding to various scales being able to recognize the second sample feature map of the scale.

[0116] In this embodiment, the recognition module may include at least one recognition layer and at least one non-recognition layer. Both the recognition layer and the non-recognition layer can be used to perform image recognition on the second sample feature map at a certain scale. However, it should be noted that after the recognition module has finished training, image recognition can be performed using only the recognition layer.

[0117] The recognition layer can include, but is not limited to, the output layer in a deep learning network model. The non-recognition layer can include, but is not limited to, the hidden layers in a deep learning network model.

[0118] The second probabilistic representation layer assigns weights to the second sample feature map at different scales for image recognition.

[0119] S14. The control recognition module performs image recognition based on the second probability corresponding to each layer in the scale-corresponding recognition module and the second sample feature map corresponding to the scale, and obtains the scale-corresponding image recognition result.

[0120] S15. If the loss function value of the identification module does not converge, adjust the parameters of the first extraction module, the coefficient determination module and the identification module, and return to step S11.

[0121] The loss function value of the recognition module represents the difference between the image recognition result corresponding to the scale and the real recognition result corresponding to the sample image.

[0122] S16. If the loss function value of the recognition module converges, the training step ends.

[0123] In this embodiment, during the training process, a pre-trained feature extraction module can be used to extract the first sample feature map corresponding to at least one sample image of a task. This eliminates the need to train the feature extraction module, reduces the parameters that need to be adjusted, and improves training efficiency.

[0124] In this embodiment, the feature extraction module, the first extraction module, the coefficient determination module, and the recognition module described above can be trained in, but are not limited to, the following ways:

[0125] S21. Obtain at least one sample image corresponding to a task.

[0126] In this step, at least one task can be referred to in the relevant introduction of step S11, and will not be repeated here.

[0127] S22. Extract the first sample feature map corresponding to the sample image based on the feature extraction module.

[0128] The first sample feature map can represent the sample image.

[0129] S23. Extract at least one sample candidate box of the first sample feature map corresponding to the sample image based on the first extraction module.

[0130] The candidate bounding boxes can be used to extract features from the feature map of the first sample.

[0131] The first extraction module may, but is not limited to, extracting at least one scale of candidate bounding boxes corresponding to the first sample feature map of the sample image based on the RPN network.

[0132] S24. Based on the candidate bounding boxes of samples at at least one scale and the first sample feature map, determine the second sample feature map at at least one scale.

[0133] In this embodiment, feature extraction can be performed on the first sample feature map based on the sample candidate boxes of each scale in at least one scale to obtain the second sample feature map of each scale, thus obtaining the second sample feature map of at least one scale.

[0134] S25. Input the second sample feature maps of various scales in at least one scale into the coefficient determination module to obtain the second probability of each layer of the recognition module corresponding to various scales being able to recognize the second sample feature map of the scale.

[0135] The higher the probability that a layer can identify the second sample feature map of a scale, the better the image recognition effect of the layer on the second sample feature map of a scale.

[0136] The second probabilistic representation layer assigns weights to the second sample feature map at different scales for image recognition.

[0137] The multiple layers of the recognition module can be found in the relevant description of step S13 above, and will not be repeated here. For example, the multiple layers of the recognition module include: output layer 11, output layer 12, output layer 13, hidden layer 21, hidden layer 22, and hidden layer 33 in the deep learning network model, such as... Figure 4 As shown, the feature extraction module can extract the first sample feature map corresponding to the sample image of each task, extract at least one sample candidate box of at least one scale corresponding to the first sample feature map of the sample image based on the first extraction module, and determine the second sample feature map of at least one scale based on the sample candidate box of at least one scale and the first sample feature map.

[0138] The second sample feature maps of various scales in at least one scale are input into the coefficient determination module to obtain the second probability (represented as k11) that the output layer 11 of the recognition module can identify the second sample feature map of the scale corresponding to various scales, the second probability (represented as k12) that the output layer 12 of the recognition module can identify the second sample feature map of the scale, the second probability (represented as k13) that the output layer 13 of the recognition module can identify the second sample feature map of the scale, the second probability (represented as k21) that the hidden layer 21 of the recognition module can identify the second sample feature map of the scale, the second probability (represented as k22) that the hidden layer 22 of the recognition module can identify the second sample feature map of the scale, and the second probability (represented as k31) that the hidden layer 31 of the recognition module can identify the second sample feature map of the scale.

[0139] c11 represents the model parameters of output layer 11, c12 represents the model parameters of output layer 12, c13 represents the model parameters of output layer 13, c21 represents the model parameters of hidden layer 21, c22 represents the model parameters of hidden layer 22, and c31 represents the model parameters of hidden layer 31.

[0140] It should be noted that, Figure 4 The identification module is just one example and is not intended to limit the identification module.

[0141] S26. The control recognition module performs image recognition based on the second probability corresponding to each layer in the scale-corresponding recognition module and the second sample feature map corresponding to the scale, and obtains the scale-corresponding image recognition result.

[0142] S27. If the loss function value of the recognition module does not converge, adjust the parameters of the feature extraction module, the first extraction module, the coefficient determination module and the recognition module, and return to step S21.

[0143] The loss function value of the recognition module represents the difference between the image recognition result corresponding to the scale and the real recognition result corresponding to the sample image.

[0144] S28. If the loss function value of the recognition module converges, the training step ends.

[0145] Understandably, after the training step is completed, the trained coefficients are used to determine the first probability that each recognition layer in the multiple recognition layers of the recognition module can recognize the second feature map of each scale based on the second feature map of at least one scale. For example, such as Figure 5 As shown, the feature extraction module extracts a first feature map corresponding to the image to be identified. Based on the first feature map, the first extraction module obtains a second feature map at at least one scale. The trained coefficient determination module determines the coefficients corresponding to each scale based on the second feature maps at various scales within the at least one scale. Figure 4 The output layers 11, 12, and 13 of the recognition module shown are configured to recognize the second feature map of scale with first probabilities k111, k112, and k113, respectively. Hidden layers 21, 22, and 31 are configured to recognize the second feature map of scale with first probabilities k211, k221, and k311, respectively. Since k111 and k113 are both less than k112, at least one layer is selected from output layers 11, 12, and 13 as the layer to be used. For example, ... Figure 5 As shown, output layer 12 is selected as the layer to be used from output layer 11, output layer 12 and output layer 13, and image recognition is performed on the second feature map of the scale based on output layer 12.

[0146] In this embodiment, the accuracy of image recognition of the second sample feature map corresponding to the scale in the corresponding layer of the recognition module depends on the training accuracy of the feature extraction module, the first extraction module, the coefficient determination module and the recognition module. The higher the training accuracy of the feature extraction module, the first extraction module, the coefficient determination module and the recognition module, the higher the accuracy of image recognition of the second sample feature map corresponding to the scale in the corresponding layer of the recognition module.

[0147] In this embodiment, by extracting candidate boxes of at least one scale corresponding to the first feature map of the image to be recognized, and based on the candidate boxes of at least one scale and the first feature map, a second feature map of at least one scale is determined. Based on the second feature maps of various scales in the at least one scale, a first probability is determined for each recognition layer in the multiple recognition layers of the recognition module corresponding to each scale to recognize the second feature map of that scale. Based on the first probability for each recognition layer to recognize the second feature map of that scale, a layer to be used is selected from the multiple recognition layers of the recognition module. This allows for the selection of a layer to be used corresponding to the scale, and image recognition is performed on the second feature map of the scale based on the layer to be used, reducing the computational load of image recognition and ensuring the accuracy of image recognition.

[0148] As another optional embodiment of this application, this embodiment is mainly a refinement of the above-mentioned coefficient determination module. The coefficient determination module may include, but is not limited to, a second extraction module, a third extraction module, and a classification module.

[0149] The second extraction module is used to extract features from the second feature maps of various scales in at least one scale to obtain the third feature maps of various scales.

[0150] In this embodiment, the second extraction module may, but is not limited to, performing convolutional operations on the second feature maps of various scales in at least one scale to obtain the third feature maps of various scales.

[0151] The third extraction module is used to extract features from the third feature maps at various scales to obtain the fourth feature maps at various scales.

[0152] In this embodiment, the third extraction module may, but is not limited to, performing global pooling layers to extract features from third feature maps of various scales to obtain fourth feature maps of various scales.

[0153] The classification module is used to determine the first probability that each recognition layer in the multiple recognition layers of the recognition module at various scales can recognize the second feature map at various scales, based on the fourth feature map at various scales.

[0154] In this embodiment, the classification module may, but is not limited to, executing the Softmax function to determine the first probability that each recognition layer in the multiple recognition layers of the recognition module corresponding to various scales can recognize the second feature map of the scale based on the fourth feature map of various scales.

[0155] like Figure 6The diagram shown is a flowchart of an image recognition method provided in the third embodiment of this application. The corresponding coefficient determination module includes an implementation of a second extraction module, a third extraction module, and a classification module. This embodiment mainly refines step S1021 in the second embodiment described above. Specifically, step S1021 may include, but is not limited to, the following steps:

[0156] Step S10211: Input the second feature maps of various scales in at least one scale into the second extraction module to obtain the third feature maps of various scales determined by the second extraction module.

[0157] The third feature map can more accurately represent the image to be identified compared to the second feature map.

[0158] Specifically, step S10211 may, but is not limited to, inputting second feature maps of various scales in at least one scale into the second extraction module, and the second extraction module performing convolutional layer convolution operations on the second feature maps of various scales to obtain third feature maps of various scales.

[0159] Step S10212: Input the third feature maps of various scales into the third extraction module to obtain the fourth feature maps of various scales determined by the third extraction module.

[0160] The fourth feature map contains fewer features that accurately represent the image to be identified compared to the third feature map.

[0161] Specifically, step S10212 may include, but is not limited to: inputting third feature maps of various scales into the third extraction module, and the third extraction module performing a global pooling layer to pool the third feature maps of various scales to obtain fourth feature maps of various scales.

[0162] The third extraction module executes a global pooling layer to pool the third feature maps at various scales, which may include, but is not limited to, max pooling or average pooling of the third feature maps at various scales.

[0163] Step S10213: Input the fourth feature maps of various scales into the classification module to obtain the first probability of each recognition layer in the multiple recognition layers of the recognition module corresponding to various scales being able to recognize the second feature map of the scale.

[0164] Specifically, step S10213 may include, but is not limited to: inputting fourth feature maps of various scales into the classification module, and the classification module executing the Softmax function to determine the first probability that each recognition layer in the multiple recognition layers of the recognition module corresponding to various scales can recognize the second feature map of the scale.

[0165] In this embodiment, second feature maps of various scales at at least one scale are input into the second extraction module to obtain third feature maps of various scales determined by the second extraction module. This ensures that the third feature maps of various scales can more accurately represent the image to be recognized. Based on this, the third feature maps of various scales are input into the third extraction module to obtain fourth feature maps of various scales determined by the third extraction module. This ensures that fewer features can be used to accurately represent the image to be recognized. The fourth feature maps of various scales are input into the classification module to obtain the first probability that each recognition layer in the multiple recognition layers of the recognition module corresponding to various scales can recognize the second feature map of the scale determined by the classification module. This ensures the accuracy and efficiency of the classification module in determining the first probability.

[0166] As another optional embodiment of this application, refer to Figure 7 This is a flowchart illustrating an image recognition method provided in the fourth embodiment of this application. This embodiment mainly refines step S103 of the first embodiment described above. Figure 7 As shown, step S103 may include, but is not limited to, the following steps:

[0167] Step S1032: Based on the hierarchical coefficient, select the top n recognition layers with the highest weights from the multiple recognition layers of the recognition module as the layers to be used, and perform image recognition on the second feature map of the scale based on the layers to be used.

[0168] In this embodiment, the weights of each recognition layer in the second feature map of the recognition module for image recognition can be sorted from largest to smallest based on the layering coefficient, and the recognition layer with the top n weights can be selected as the layer to be used.

[0169] In this application, n can be set as needed, and the value of n is not restricted in this application.

[0170] When there are multiple layers to be used, image recognition can be performed on the second feature map of each layer at each scale to obtain the image recognition result corresponding to each layer.

[0171] In this embodiment, the average calculation can be performed on the image recognition results corresponding to each layer to be used to obtain the image recognition results to be used.

[0172] In this embodiment, a second feature map of at least one scale is obtained based on a first feature map corresponding to the image to be recognized. Based on the second feature maps of various scales in the at least one scale, the layering coefficients corresponding to various scales are determined. Based on the layering coefficients, the recognition layers with the top n weights are selected from multiple recognition layers of the recognition module as the layers to be used. This can realize the selection of multiple layers to be used corresponding to the scale, so as to perform image recognition based on the second feature map of each scale based on each layer to be used, thereby reducing the amount of computation for image recognition and ensuring the accuracy of image recognition.

[0173] As another optional embodiment of this application, refer to Figure 8 This is a flowchart illustrating an image recognition method provided in the fifth embodiment of this application. This embodiment mainly refines step S103 of the first embodiment described above. Figure 8 As shown, step S103 may include, but is not limited to, the following steps:

[0174] Step S1033: Based on the hierarchical coefficient, select the recognition layer with the largest weight from multiple recognition layers of the recognition module as the layer to be used, and perform image recognition based on the second feature map of the scale based on the layer to be used.

[0175] In this embodiment, the image recognition weights of the second feature maps of the scale in the multiple recognition layers of the recognition module can be sorted based on the layering coefficient, and the recognition layer with the largest weight can be selected as the layer to be used from the multiple recognition layers of the recognition module.

[0176] By selecting the recognition layer with the highest weight from multiple recognition layers in the recognition module as the layer to be used, and performing image recognition on the second feature map of the scale based on the layer to be used, the accuracy of image recognition can be guaranteed.

[0177] In this embodiment, a second feature map of at least one scale is obtained based on a first feature map corresponding to the image to be recognized. Based on the second feature maps of various scales in the at least one scale, the layering coefficients corresponding to various scales are determined. Based on the layering coefficients, the recognition layer with the largest weight is selected from multiple recognition layers of the recognition module as the layer to be used. This can realize the selection of the layer to be used corresponding to the scale, so as to perform image recognition based on the second feature map of the scale based on the layer to be used, thereby reducing the amount of computation for image recognition and ensuring the accuracy of image recognition.

[0178] The following section describes an image recognition device provided in this application. The image recognition device described below can be referred to in correspondence with the image recognition method described above.

[0179] Please see Figure 9 The image recognition device includes: an acquisition unit 100, a determination unit 200, and a selection unit 300.

[0180] The obtaining unit 100 is used to obtain a second feature map of at least one scale based on a first feature map corresponding to the image to be identified.

[0181] The determining unit 200 is used to determine the layering coefficients corresponding to various scales based on the second feature maps of various scales in at least one scale. The layering coefficients are used to characterize the weight of each recognition layer in the multiple recognition layers of the recognition module for image recognition of the second feature maps of various scales. The recognition layers are used to perform image recognition on the second feature maps of various scales.

[0182] Selection unit 300 is used to select a layer to be used from multiple recognition layers of the recognition module based on the hierarchical coefficient, so as to perform image recognition on the second feature map of the scale based on the layer to be used.

[0183] The image recognition device can be executed based on a multi-task autonomous hierarchical model, which includes a feature extraction module, a hierarchical estimation module, and a recognition module.

[0184] The feature extraction module is used to extract the first feature map corresponding to the image to be identified;

[0185] The hierarchical estimation module is used to obtain a second feature map of at least one scale based on a first feature map corresponding to the image to be identified, and to determine the hierarchical coefficients corresponding to each scale based on the second feature maps of various scales in the at least one scale.

[0186] The recognition module is used to select a layer to be used from multiple recognition layers of the recognition module based on the hierarchical coefficient, so as to perform image recognition on the second feature map of the scale based on the layer to be used.

[0187] The hierarchical estimation module includes: a first extraction module and a coefficient determination module;

[0188] The first extraction module is used to extract candidate boxes of at least one scale corresponding to the first feature map of the image to be identified; and,

[0189] Based on candidate boxes of at least one scale and the first feature map, a second feature map of at least one scale is determined;

[0190] The coefficient determination module is used to determine the first probability that each recognition layer in the multiple recognition layers of the recognition module corresponding to each scale can recognize the second feature map of the scale based on the second feature map of each scale in at least one scale. The first probability represents the weight of the recognition layer for image recognition of the second feature map of the scale.

[0191] The coefficient determination module may include: a second extraction module, a third extraction module, and a classification module;

[0192] Based on second feature maps of various scales within at least one scale, determining the first probability that each recognition layer in multiple recognition layers of the recognition module corresponding to each scale can recognize the second feature map of that scale can include:

[0193] Input the second feature maps of various scales in at least one scale into the second extraction module to obtain the third feature maps of various scales determined by the second extraction module;

[0194] The third feature maps at various scales are input into the third extraction module to obtain the fourth feature maps at various scales determined by the third extraction module.

[0195] By inputting the fourth feature maps at various scales into the classification module, the first probability of each recognition layer in the recognition module that can recognize the second feature map at each scale is obtained, as determined by the classification module.

[0196] Select unit 300, which can be specifically used for:

[0197] Based on the hierarchical coefficient, the nth recognition layer with the highest weight among the multiple recognition layers of the recognition module is selected as the layer to be used;

[0198] or,

[0199] Based on the hierarchical coefficient, the recognition layer with the highest weight is selected as the layer to be used from multiple recognition layers of the recognition module.

[0200] In this embodiment, the image recognition device may further include a first training unit, used to train the first extraction module, the coefficient determination module, and the recognition module in the following manner:

[0201] Obtain the first sample feature map corresponding to at least one sample image for a task;

[0202] Based on the first extraction module, extract a second sample feature map of at least one scale corresponding to the first sample feature map of the sample image;

[0203] The second sample feature maps of various scales in at least one scale are input into the coefficient determination module to obtain the second probability of each layer in the recognition module corresponding to various scales that can recognize the second sample feature map of the scale. The second probability characterization layer performs image recognition on the second sample feature map of the scale.

[0204] The control recognition module performs image recognition based on the second probability corresponding to each layer in the scale-corresponding recognition module and the second sample feature map corresponding to the scale, and obtains the scale-corresponding image recognition result.

[0205] If the loss function value of the recognition module does not converge, adjust the parameters of the first extraction module, the coefficient determination module and the recognition module, and return to the step of obtaining the first sample feature map corresponding to at least one sample image corresponding to the task. The loss function value of the recognition module represents the difference between the image recognition result corresponding to the scale and the real recognition result corresponding to the sample image.

[0206] If the loss function value of the recognition module converges, the training step ends.

[0207] In this embodiment, the image recognition device may further include a second training unit, used to train the feature extraction module, the first extraction module, the coefficient determination module, and the recognition module in the following manner:

[0208] Obtain at least one sample image corresponding to a task;

[0209] The first sample feature map corresponding to the sample image is extracted based on the feature extraction module;

[0210] Based on the first extraction module, at least one scale of candidate bounding boxes corresponding to the first sample feature map of the sample image are extracted;

[0211] Based on the candidate bounding boxes of at least one scale and the first sample feature map, a second sample feature map of at least one scale is determined.

[0212] The second sample feature maps of various scales in at least one scale are input into the coefficient determination module to obtain the second probability of each layer of the recognition module corresponding to various scales that can recognize the second sample feature map of the scale determined by the coefficient determination module. The second probability characterization layer performs image recognition on the second sample feature map of the scale.

[0213] The control recognition module performs image recognition based on the second probability corresponding to each layer in the scale-corresponding recognition module and the second sample feature map corresponding to the scale, and obtains the scale-corresponding image recognition result.

[0214] If the loss function value of the recognition module does not converge, adjust the parameters of the feature extraction module, the first extraction module, the coefficient determination module and the recognition module, and return to the step of obtaining at least one sample image corresponding to the task. The loss function value of the recognition module represents the difference between the image recognition result corresponding to the scale and the real recognition result corresponding to the sample image.

[0215] If the loss function value of the recognition module converges, the training step ends.

[0216] Corresponding to the above-described embodiment of an image recognition method provided in this application, this application also provides an embodiment of an electronic device that applies the image recognition method.

[0217] The electronic device may include the following structure:

[0218] Memory and processor.

[0219] Memory, used to store at least one set of instructions;

[0220] The processor is used to call and execute the instruction set in memory, and execute the image recognition method as described in any one of the first to fifth embodiments above.

[0221] Corresponding to the image recognition method embodiment provided in this application above, this application also provides an embodiment of a storage medium.

[0222] In this embodiment, the storage medium stores a computer program that implements the image recognition method described in any one of the first to fifth embodiments. The computer program is executed by a processor to implement the steps of the image recognition method described in any one of the first to fifth embodiments.

[0223] It should be noted that each embodiment focuses on describing the differences from other embodiments, and the same or similar parts between the embodiments can be referred to accordingly. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments.

[0224] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0225] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0226] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0227] The above provides a detailed description of the image recognition method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image recognition method, comprising: Based on the first feature map corresponding to the image to be identified, a second feature map of at least one scale is obtained; Based on the second feature maps of various scales in the at least one scale, a layering coefficient corresponding to each scale is determined. The layering coefficient is used to characterize the first probability that each recognition layer in the multiple recognition layers of the recognition module corresponding to each scale can recognize the second feature map of the scale. The first probability characterizes the weight of the recognition layer for image recognition of the second feature map of the scale. The recognition layer is used for image recognition of the second feature map of the scale. Based on the layering coefficient, a layer to be used is selected from multiple recognition layers of the recognition module to perform image recognition on the second feature map of the scale based on the layer to be used, and the weight of the layer to be used for image recognition on the second feature map of the scale satisfies the set weight conditions.

2. The method according to claim 1, wherein the image recognition method is executed based on a multi-task autonomous hierarchical model, the multi-task autonomous hierarchical model comprising: Feature extraction module, hierarchical estimation module, and recognition module; The feature extraction module is used to extract the first feature map corresponding to the image to be identified; The hierarchical estimation module is used to obtain a second feature map of at least one scale based on a first feature map corresponding to the image to be identified, and to determine the hierarchical coefficients corresponding to each scale based on the second feature maps of each scale among the at least one scale. The recognition module is used to select a layer to be used from multiple recognition layers of the recognition module based on the layering coefficient, so as to perform image recognition on the second feature map of the scale based on the layer to be used.

3. The method according to claim 2, wherein the hierarchical estimation module comprises: The first extraction module and the coefficient determination module; The first extraction module is used to extract candidate boxes of at least one scale corresponding to the first feature map of the image to be identified; as well as, Based on the candidate boxes of at least one scale and the first feature map, a second feature map of at least one scale is determined; The coefficient determination module is used to determine, based on the second feature maps of various scales in the at least one scale, the first probability that each recognition layer of the recognition module corresponding to each scale can recognize the second feature map of the scale.

4. The method according to claim 3, wherein the coefficient determination module comprises: The module consists of a second extraction module, a third extraction module, and a classification module. Based on the second feature maps of various scales in the at least one scale, determining the first probability that each recognition layer in the multiple recognition layers of the recognition module corresponding to each scale can recognize the second feature map of the scale includes: The second feature maps of various scales in the at least one scale are input into the second extraction module to obtain the third feature maps of various scales determined by the second extraction module; The third feature maps of various scales are input into the third extraction module to obtain the fourth feature maps of various scales determined by the third extraction module. The fourth feature maps of various scales are input into the classification module to obtain the first probability that each of the multiple recognition layers of the recognition module corresponding to the various scales can recognize the second feature map of the scale.

5. The method according to claim 1, wherein selecting a layer to be used from a plurality of recognition layers of the recognition module based on the layering coefficient, includes: Based on the layering coefficient, the recognition layer with the top n weights from the multiple recognition layers of the recognition module is selected as the layer to be used; or, Based on the layering coefficient, the layer with the highest weight is selected from the multiple recognition layers of the recognition module as the layer to be used.

6. The method according to claim 3, wherein the first extraction module, the coefficient determination module, and the recognition module are trained in the following manner: Obtain the first sample feature map corresponding to at least one sample image for a task; Based on the first extraction module, extract at least one scale of second sample feature map corresponding to the first sample feature map of the sample image; The second sample feature maps of various scales in the at least one scale are input into the coefficient determination module to obtain the second probability that each layer of the recognition module corresponding to various scales can recognize the second sample feature map of the scale, as determined by the coefficient determination module. The second probability represents the weight of the layer for image recognition of the second sample feature map of the scale. The recognition module is controlled to perform image recognition based on the second probability corresponding to each layer in the recognition module corresponding to the scale and the second sample feature map corresponding to the scale, so as to obtain the image recognition result corresponding to the scale. If the loss function value of the recognition module does not converge, the parameters of the first extraction module, the coefficient determination module and the recognition module are adjusted, and the step of obtaining the first sample feature map corresponding to the sample image corresponding to at least one task is returned. The loss function value of the recognition module represents the difference between the image recognition result corresponding to the scale and the real recognition result corresponding to the sample image. If the loss function value of the recognition module converges, the training step ends.

7. The method according to claim 3, wherein the feature extraction module, the first extraction module, the coefficient determination module, and the recognition module are trained in the following manner: Obtain at least one sample image corresponding to a task; Based on the feature extraction module, a first sample feature map corresponding to the sample image is extracted; Based on the first extraction module, at least one scale of candidate bounding boxes corresponding to the first sample feature map of the sample image are extracted; Based on the candidate bounding boxes of the samples at the at least one scale and the first sample feature map, a second sample feature map of at least one scale is determined; The second sample feature maps of various scales in the at least one scale are input into the coefficient determination module to obtain the second probability that each layer of the recognition module corresponding to various scales can recognize the second sample feature map of the scale, as determined by the coefficient determination module. The second probability represents the weight of the layer for image recognition of the second sample feature map of the scale. The recognition module is controlled to perform image recognition based on the second probability corresponding to each layer in the recognition module corresponding to the scale and the second sample feature map corresponding to the scale, so as to obtain the image recognition result corresponding to the scale. If the loss function value of the recognition module does not converge, the parameters of the feature extraction module, the first extraction module, the coefficient determination module and the recognition module are adjusted, and the step of obtaining at least one sample image corresponding to the task is returned. The loss function value of the recognition module represents the difference between the image recognition result corresponding to the scale and the real recognition result corresponding to the sample image. If the loss function value of the recognition module converges, the training step ends.

8. An image recognition device, comprising: The obtaining unit is used to obtain a second feature map of at least one scale based on a first feature map corresponding to the image to be identified; A determining unit is configured to determine a layering coefficient corresponding to each of the at least one scale based on the second feature maps of each of the various scales. The layering coefficient is used to characterize the first probability that each of the multiple recognition layers of the recognition module corresponding to each of the various scales can recognize the second feature map of the scale. The first probability characterizes the weight of the recognition layer for image recognition of the second feature map of the scale. The recognition layer is used for image recognition of the second feature map of the scale. The selection unit is used to select a layer to be used from multiple recognition layers of the recognition module based on the hierarchical coefficient, so as to perform image recognition on the second feature map of the scale based on the layer to be used, wherein the weight of the layer to be used for image recognition on the second feature map of the scale satisfies the set weight condition.

9. An electronic device, comprising: Memory and processor; The memory is used to store at least one set of instructions; The processor is configured to call and execute the instruction set in the memory, and execute the image recognition method as described in any one of claims 1-7 by executing the instruction set.

10. A storage medium storing a computer program that implements the image recognition method as described in any one of claims 1-7, the computer program being executed by a processor to implement the image recognition method as described in any one of claims 1-7.

Images