Image recognition methods, systems, media, and devices based on neural networks
By using a lightweight neural network to assist a high-precision neural network and employing multi-layer information fusion to predict the exit layer, the problem of high energy consumption of deep learning models on resource-constrained systems is solved, achieving more efficient image recognition and resource utilization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2023-03-24
- Publication Date
- 2026-06-02
AI Technical Summary
In existing technologies, when deep learning models are deployed on resource-constrained systems, they consume a lot of energy and cannot effectively predict premature exit points, resulting in a waste of computing resources.
A lightweight neural network is used to assist a high-precision neural network. By fusing information from multiple layers, the exit layer is predicted. The lightweight neural network is used to predict the exit point of the high-precision neural network in advance, thereby reducing the amount of computation and energy consumption.
It improves the prediction accuracy of image recognition, reduces computational load and energy consumption, and achieves more efficient resource utilization.
Smart Images
Figure CN116229184B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neural network processing technology, specifically to image recognition methods, systems, media, and devices based on neural networks, and more specifically to image recognition methods and systems that utilize extremely lightweight neural networks to assist high-precision neural networks. Background Technology
[0002] Deep learning methods, such as convolutional neural networks, have achieved great success in versatile applications. However, one of the challenges of deploying deep learning models on resource-constrained systems is their significant energy cost. Early exit, as a dynamic inference method, adds an exit layer to the network, which can terminate inference early and obtain accurate results to save energy. Current passive decision-making for energy regulation used for early exit cannot adapt to the ongoing inference state, varying inference workloads, and time constraints, let alone guide the rational configuration of the computing platform during inference to save potential energy.
[0003] Patent document US16997885 (application number: 20210056357A1) discloses a system and method for implementing flexible, input-adaptive deep learning neural networks; patent document US16509098 (application number: 20210012178A1) discloses a system, method, and apparatus for early exit from convolution; the above-mentioned prior art proposes methods for early exit, but cannot predict the early exit point; Summary of the Invention
[0004] In view of the deficiencies in the prior art, the purpose of this invention is to provide an image recognition method, system, medium and device based on neural networks.
[0005] According to the present invention, an image recognition method based on a high-precision neural network includes: obtaining an image recognition exit layer using a lightweight neural network; using the current image recognition exit layer as the predicted early exit point of the high-precision neural network; activating the exit layer of the network layer corresponding to the early exit point based on the early exit point; and obtaining the current image recognition result.
[0006] Preferably, the image to be recognized is input into a lightweight neural network to obtain an image recognition exit layer; or the image to be recognized is input into a high-precision neural network, and the recognition result of a certain network layer is input into a lightweight neural network to obtain an image recognition exit layer; or the image to be recognized is input into a high-precision neural network, and the recognition results of multiple consecutive network layers are fused and calculated, and the fused calculation result is input into a lightweight neural network to obtain an image recognition exit layer.
[0007] Preferably, the fusion calculation of the identification results of multiple consecutive network layers is performed by weighted summation of the preset network layer identification results to obtain the weighted summation result.
[0008] Preferably, the high-precision neural network includes: a CNN convolutional neural network and a Transformer neural network.
[0009] According to the present invention, an image recognition system based on a high-precision neural network includes: obtaining an image recognition exit layer using a lightweight neural network; using the current image recognition exit layer as the predicted early exit point of the high-precision neural network; activating the exit layer of the network layer corresponding to the early exit point based on the early exit point; and obtaining the current image recognition result.
[0010] Preferably, the image to be recognized is input into a lightweight neural network to obtain an image recognition exit layer; or the image is input into a high-precision neural network, and the recognition result of a certain network layer is input into a lightweight neural network to obtain an image recognition exit layer; or the image is input into a high-precision neural network, and the recognition results of multiple consecutive network layers are fused and calculated, and the fused calculation result is input into a lightweight neural network to obtain an image recognition exit layer.
[0011] Preferably, the fusion calculation of the identification results of multiple consecutive network layers is performed by weighted summation of the preset network layer identification results to obtain the weighted summation result.
[0012] Preferably, the high-precision neural network includes: a CNN convolutional neural network and a Transformer neural network.
[0013] According to the present invention, a computer-readable storage medium storing a computer program is provided, wherein the computer program, when executed by a processor, implements the steps of the method described above.
[0014] An electronic device according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the computer program, when executed by the processor, implements the steps of the image recognition method based on a high-precision neural network described above.
[0015] Compared with the prior art, the present invention has the following beneficial effects:
[0016] 1. This invention improves prediction accuracy by fusing information from the multi-layer prediction architecture of the multi-layer intermediate results of a high-precision neural network with the multi-layer calculation results.
[0017] 2. This invention utilizes a lightweight neural network for image recognition based on fused information. It first obtains the exit layer, and then predicts the exit layer of the high-precision neural network based on the exit layer obtained by the lightweight neural network, thereby reducing the cost of each exit layer execution by the high-precision neural network, and reducing computational load, power consumption and energy consumption. Attached Figure Description
[0018] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0019] Figure 1 This is a schematic diagram of the exit structure for neural network inference prediction based on multi-layer information fusion.
[0020] Figure 2 This is a schematic diagram of the exit structure for neural network inference prediction based on multi-layer information fusion.
[0021] Figure 3 This is a schematic diagram of the exit structure for neural network inference prediction based on multi-layer information fusion. Detailed Implementation
[0022] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.
[0023] Example 1
[0024] In the prior art, when using high-precision neural networks (i.e. large neural networks) for image recognition, the cost of each execution of the exit layer is large, and the accuracy is low. In order to overcome the technical problems of low accuracy and high cost in traditional methods when predicting with large networks, this invention provides an image recognition method and system that uses an extremely lightweight neural network to assist a high-precision neural network.
[0025] The present invention provides an image recognition method using an extremely lightweight neural network to assist a high-precision neural network, comprising: obtaining an image recognition exit layer using a lightweight neural network; using the current image recognition exit layer as the prediction recognition early exit point of the high-precision neural network; activating the exit layer of the network layer corresponding to the early exit point based on the early exit point; and obtaining the current image recognition result.
[0026] The image to be recognized can be input into a lightweight neural network to obtain an image recognition exit layer; or the image to be recognized can be input into a high-precision neural network, and the recognition result of a certain network layer can be input into a lightweight neural network to obtain an image recognition exit layer; or the image to be recognized can be input into a high-precision neural network, and the recognition results of multiple consecutive network layers can be fused and calculated, and the fused calculation result can be input into a lightweight neural network to obtain an image recognition exit layer.
[0027] For example: directly input the image to be recognized into the lightweight neural network; or first input the image to be recognized into the high-precision neural network, and then input the recognition result of the first network layer of the high-precision neural network into the lightweight neural network; or first input the image to be recognized into the high-precision neural network, and then perform fusion calculation on the recognition results of the first network layer, the second network layer, and the third network layer of the high-precision neural network, and then input the fusion calculation result into the lightweight neural network.
[0028] Specifically, the fusion calculation of the identification results of multiple consecutive network layers (e.g., the calculation results of layers 1, 2, and 3) is performed by weighting and summing the pre-defined network layer identification results to obtain the weighted sum. This multi-layer prediction structure and information fusion prediction method can achieve higher prediction accuracy.
[0029] Specifically, the fusion calculation results are used to obtain the recognition exit point through a lightweight neural network, and the network layer corresponding to the current recognition exit point is used as the recognition exit point predicted by the high-precision neural network. For example, if the image recognition exit layer is obtained as the 11th layer through the lightweight neural network, then the prediction high-precision neural network will also exit at the 11th layer. When the high-precision neural network exits at the 11th layer and the image recognition does not meet the preset requirements, the high-precision neural network continues to perform image recognition until the obtained recognition result meets the preset requirements.
[0030] Specifically, the high-precision neural network includes: CNN convolutional neural network and Transformer neural network.
[0031] Example 2
[0032] Example 2 is a preferred example of Example 1.
[0033] An image recognition system using an extremely lightweight neural network to assist a high-precision neural network, provided by the present invention, includes: obtaining an image recognition exit layer using a lightweight neural network; using the current image recognition exit layer as the predicted early exit point of the high-precision neural network; activating the exit layer of the network layer corresponding to the early exit point based on the early exit point; and obtaining the current image recognition result.
[0034] The image to be recognized can be input into a lightweight neural network to obtain an image recognition exit layer; or the image to be recognized can be input into a high-precision neural network, and the recognition result of a certain network layer can be input into a lightweight neural network to obtain an image recognition exit layer; or the image to be recognized can be input into a high-precision neural network, and the recognition results of multiple consecutive network layers can be fused and calculated, and the fused calculation result can be input into a lightweight neural network to obtain an image recognition exit layer.
[0035] For example: Figures 1 to 3 As shown, the image to be recognized can be directly input into a lightweight neural network; or the image to be recognized can be first input into a high-precision neural network, and then the recognition result of the first network layer of the high-precision neural network can be input into the lightweight neural network; or the image to be recognized can be first input into a high-precision neural network, and then the recognition results of the first network layer, the second network layer, and the third network layer of the high-precision neural network can be fused and calculated, and then the fused calculation result can be input into the lightweight neural network.
[0036] Specifically, the fusion calculation of the identification results of multiple consecutive network layers (e.g., the calculation results of layers 1, 2, and 3) is performed by weighting and summing the pre-defined network layer identification results to obtain the weighted sum. This multi-layer prediction structure and information fusion prediction method can achieve higher prediction accuracy.
[0037] Specifically, the fusion calculation results are used to obtain the recognition exit point through a lightweight neural network, and the network layer corresponding to the current recognition exit point is used as the recognition exit point predicted by the high-precision neural network. For example, if the image recognition exit layer is obtained as the 11th layer through the lightweight neural network, then the prediction high-precision neural network will also exit at the 11th layer. When the high-precision neural network exits at the 11th layer and the image recognition does not meet the preset requirements, the high-precision neural network continues to perform image recognition until the obtained recognition result meets the preset requirements.
[0038] Specifically, the high-precision neural network includes: CNN convolutional neural network and Transformer neural network.
[0039] It should be noted that those skilled in the art can make several changes and improvements without departing from the inventive concept; at the same time, the inventive concept of the present invention is also applicable to voice and text recognition; these all fall within the protection scope of the present invention.
[0040] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.
[0041] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A high-precision neural network-based image recognition method, characterized by, include: Obtaining the image recognition exit layer using a lightweight neural network; The current image recognition exit layer is used as the prediction and recognition early exit point of the high-precision neural network. Based on the early exit point, the exit layer of the network layer corresponding to the early exit point is activated to obtain the current image recognition result. The image to be recognized is input into a high-precision neural network, and the recognition results of multiple consecutive network layers are fused and calculated. The fused calculation result is then input into a lightweight neural network to obtain the image recognition exit layer. The method for fusing and calculating the identification results of multiple consecutive network layers is as follows: weighted summation of the identification results of the preset network layers is performed to obtain the weighted summation result. 2.The high-precision neural network-based image recognition method of claim 1, wherein, The high-precision neural networks include: CNN convolutional neural networks and Transformer neural networks.
3. A high-precision neural network-based image recognition system, characterized by, include: Obtaining the image recognition exit layer using a lightweight neural network; The current image recognition exit layer is used as the prediction and recognition early exit point of the high-precision neural network. Based on the early exit point, the exit layer of the network layer corresponding to the early exit point is activated to obtain the current image recognition result. The image to be recognized is input into a high-precision neural network, and the recognition results of multiple consecutive network layers are fused and calculated. The fused calculation result is then input into a lightweight neural network to obtain the image recognition exit layer. The method for fusing and calculating the identification results of multiple consecutive network layers is as follows: weighted summation of the identification results of the preset network layers is performed to obtain the weighted summation result.
4. The image recognition system based on a high-precision neural network according to claim 3, characterized in that, The high-precision neural networks include: CNN convolutional neural networks and Transformer neural networks.
5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 2.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of the image recognition method based on a high-precision neural network as described in any one of claims 1 to 2.