Self-powered visual perception system, energy scheduling method, device and electronic equipment

Abstract

The present invention provides a self-powered visual perception system, an energy scheduling method, a device and an electronic device, wherein the self-powered visual perception system is applied to operate an image recognition neural network, and the system comprises: a pixel array, an in-memory computing unit, a power management unit and an energy scheduling module, wherein the power management unit is electrically connected to the pixel array, and is used to boost the energy collected by the pixel array, and store the processed energy in an energy storage capacitor; the energy scheduling module is communicatively connected to the energy storage capacitor, the pixel array and the in-memory computing unit, and is used to monitor the energy stored in the energy storage capacitor in real time, and based on the monitoring results, the pixel array and / or the in-memory computing unit are called to calculate the image recognition neural network. The present invention realizes the ability to support the operation of the image recognition neural network and to self-power in situ, thereby improving energy utilization.

Description

Technical Field

[0001] The present invention relates to the field of visual perception technology, and in particular to a self-powered visual perception system, an energy scheduling method, a device and an electronic device. Background Art

[0002] In the era of the continuous development of IoT technology, intelligent vision systems are widely deployed at the edge and fully applied in the field of perception tasks. Traditional intelligent edge devices are mostly powered by external batteries, but the limited energy resources stored in the battery limit the working life of these intelligent edge devices. In order to extend the working life of battery-powered intelligent edge devices, energy harvesting modules are often added to the intelligent perception system or low-power perception circuits are used.

[0003] It is known from relevant technologies that in terms of energy collection, in-situ energy collection can be used. By taking advantage of the characteristics of sensors as energy converters, image sensors, which are essential in visual perception systems, can be used to collect energy, making it possible to realize self-powered miniaturized perception systems. In terms of low-power perception design, low-power processing architectures can be used to enable perception systems to work under energy-constrained conditions, laying the foundation for realizing in-situ self-powered intelligent visual perception systems.

[0004] However, the research work on in-situ energy harvesting and low-power sensing circuit design is carried out separately. Even with the dual features of in-situ energy harvesting and intelligent sensing, it can only achieve simple motion detection tasks. Currently, finding a design solution for an in-situ self-powered system that supports running more complex algorithms has become a hot topic in current research. Summary of the invention

[0005] The present invention provides a self-powered visual perception system, an energy scheduling method, a device and an electronic device, which can support the operation of an image recognition neural network and self-power in situ, thereby improving energy utilization.

[0006] The present invention provides a self-powered visual perception system, which is used to perform operations on an image recognition neural network. The system includes: a pixel array, an in-memory computing unit, a power management unit and an energy scheduling module, wherein the power management unit is electrically connected to the pixel array, and is used to boost the energy collected by the pixel array, and store the processed energy in an energy storage capacitor; the energy scheduling module is respectively communicated with the energy storage capacitor, the pixel array and the in-memory computing unit, and is used to monitor the energy stored in the energy storage capacitor in real time, and call the pixel array and\or the in-memory computing unit to perform calculations on the image recognition neural network based on the monitoring results; wherein the pixel array and the in-memory computing unit are respectively electrically connected to the energy storage capacitor, the pixel array calculates the first layer network of the image recognition neural network based on the energy stored in the energy storage capacitor, and the in-memory computing unit calculates the auxiliary layer network of the image recognition neural network based on the energy stored in the energy storage capacitor, wherein the auxiliary layer network is the other layer network of the image recognition neural network except the first layer network.

[0007] According to a self-powered visual perception system provided by the present invention, the self-powered visual perception system also includes an activation value cache module, wherein the activation value cache module is electrically connected to the pixel array and the in-memory computing unit, respectively, and is used to store calculation results of the pixel array and the in-memory computing unit.

[0008] According to a self-powered visual perception system provided by the present invention, the self-powered visual perception system also includes a read-only memory and a startup module, wherein the startup module is electrically connected to the energy storage capacitor, and is used for starting based on the energy stored in the energy storage capacitor and sending a startup signal; the read-only memory is communicatively connected to the startup module, and is used for loading the stored weight data of the first layer network of the image recognition neural network into the pixel array when the read-only memory receives the startup signal sent by the startup module, so that the pixel array calculates the first layer network based on the weight data of the first layer network, and loading the stored weight data of the auxiliary layer network of the image recognition neural network into the in-memory computing unit, so that the in-memory computing unit calculates the auxiliary layer network based on the weight data of the auxiliary layer network.

[0009] According to a self-powered visual perception system provided by the present invention, the in-memory computing unit includes multiple in-memory computing sub-units, wherein the in-memory computing sub-units respectively correspond to each auxiliary layer network of the image recognition neural network, and perform calculations on each corresponding auxiliary layer network.

[0010] According to a self-powered visual perception system provided by the present invention, the energy scheduling module includes a logic control unit and a plurality of level detection units, the level detection units respectively corresponding to the pixel array and the in-memory computing subunit, wherein the level detection unit is used to monitor in real time whether the energy stored in the energy storage capacitor meets the execution start threshold, wherein the execution start threshold is the minimum energy value required for the pixel array and the in-memory computing subunit to operate; the logic control unit is used to call the pixel array to calculate the first layer network of the image recognition neural network, and\or call the in-memory computing subunit to calculate the auxiliary layer network of the image recognition neural network.

[0011] The present invention also provides an energy scheduling method, which is applied to a self-powered visual perception system, and the method includes: real-time monitoring of the energy stored in an energy storage capacitor, wherein the energy stored in the energy storage capacitor is obtained based on the energy used by a pixel array; based on the monitoring results, calling the pixel array to calculate the first layer network of the image recognition neural network, and\or calling an in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network, wherein the auxiliary layer network is other layer networks of the image recognition neural network except the first layer network; based on the calculation results of the pixel array and the calculation results of the in-memory computing unit, obtaining the recognition result of the image recognition neural network.

[0012] According to the energy scheduling method provided by the present invention, the in-memory computing unit includes multiple in-memory computing sub-units, wherein the in-memory computing sub-units correspond to each auxiliary layer network respectively; the calling of the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network specifically includes: calling the in-memory computing sub-unit to calculate the auxiliary layer network of the image recognition neural network corresponding to the in-memory computing sub-unit.

[0013] According to the energy scheduling method provided by the present invention, based on the monitoring results, calling the pixel array to calculate the first layer network of the image recognition neural network, and\or calling the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network, specifically includes: if it is monitored that the energy stored in the energy storage capacitor meets the execution start threshold, calling the pixel array to calculate the first layer network of the image recognition neural network, and\or calling the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network.

[0014] According to the energy scheduling method provided by the present invention, the execution start threshold includes a first execution start threshold and a second execution start threshold; if it is monitored that the energy stored in the energy storage capacitor meets the execution start threshold, the pixel array is called to calculate the first layer network of the image recognition neural network, and\or the in-memory calculation unit is called to calculate the auxiliary layer network of the image recognition neural network, specifically including: if it is monitored that the energy stored in the energy storage capacitor meets the first execution start threshold, the pixel array is called to calculate the first layer network of the image recognition neural network, wherein the first execution start threshold is the minimum energy value required for the pixel array to operate; if it is monitored that the energy stored in the energy storage capacitor meets the second execution start threshold, the in-memory calculation subunit is called to calculate the auxiliary layer network of the image recognition neural network, wherein the second execution start threshold is the minimum energy value required for the in-memory calculation subunit to operate.

[0015] According to the energy scheduling method provided by the present invention, before calling the pixel array to calculate the first layer network of the image recognition neural network based on the monitoring results, and\or calling the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network, the method also includes: if it is monitored that the energy stored in the energy storage capacitor does not meet the execution start threshold, energy is collected based on the pixel array.

[0016] According to the energy scheduling method provided by the present invention, the execution start threshold is determined in the following manner: respectively determining the error tolerance of each layer network in the image recognition neural network; based on the error tolerance, determining the lower limit power supply voltage of each layer network, wherein the lower limit power supply voltage corresponds to the error tolerance; based on the lower limit power supply voltage, determining the capacitance value of the energy storage capacitor; based on the capacitance value of the energy storage capacitor and the lower limit power supply voltage corresponding to the layer network, determining the execution start threshold corresponding to the layer network.

[0017] According to the energy scheduling method provided by the present invention, the error tolerance is determined in the following manner: determining an initial image recognition neural network; introducing a computational error into each initial layer network of the initial image recognition neural network, wherein the computational error is obtained by binary search, so that the inference accuracy of the initial image recognition neural network into which the computational error is inserted reaches an accuracy threshold after layer-by-layer retraining, wherein the accuracy threshold is determined based on the image recognition neural network; and using the error value of the computational error as the error tolerance.

[0018] According to the energy scheduling method provided by the present invention, the introducing of computational errors to each initial layer network of the initial image recognition neural network specifically includes: determining the single-layer error tolerance value and the single-layer calculation amount of each initial layer network of the initial image recognition neural network based on the initial image recognition neural network, wherein the single-layer error tolerance value is the maximum error rate that the initial layer network can bear when the inference accuracy reaches the accuracy threshold when the computational error is introduced; calculating the retraining factor of each initial layer network of the initial image recognition neural network based on the single-layer error tolerance value and the single-layer calculation amount; sorting the retraining factors in descending order to obtain an error injection order; and introducing computational errors to each initial layer network of the initial image recognition neural network according to the error injection order.

[0019] According to the energy scheduling method provided by the present invention, the capacitance value of the energy storage capacitor is determined based on the lower limit power supply voltage, specifically including: determining the standard operating voltage of each layer of the network in the image recognition neural network; determining the supply capacitance value required for the layer network based on the lower limit power supply voltage corresponding to the layer network and the standard operating voltage; and using the largest supply capacitance value among the supply capacitance values ​​as the capacitance value of the energy storage capacitor.

[0020] According to the energy scheduling method provided by the present invention, the supply capacitance value required by the layer network is determined based on the lower limit power supply voltage corresponding to the layer network and the standard operating voltage, and is implemented by the following formula:

[0021]

[0022] Among them, C ST,i represents the supply capacitance value required for the i-th layer network in the image recognition neural network; V max Indicates the standard operating voltage; V Li represents the lower limit power supply voltage corresponding to the i-th layer network; E f,i (g(V max ,V Li )) indicates that at a fixed working voltage g(V max ,V Li ) to calculate the energy consumption value required by the i-th layer network, where g(V max ,V Li ) is determined using the following formula:

[0023]

[0024] According to the energy scheduling method provided by the present invention, the execution start threshold corresponding to the layer network is determined based on the capacitance value of the energy storage capacitor and the lower limit power supply voltage corresponding to the layer network, and is implemented by the following formula:

[0025]

[0026] Among them, V Hi represents the execution start threshold corresponding to the i-th layer network; C ST Represents the capacitance value of the energy storage capacitor; V Li represents the lower limit power supply voltage corresponding to the i-th layer network; E f,i (g(V Hi ,V Li )) indicates that at a fixed working voltage g(V Hi ,V Li ) to calculate the energy consumption value required by the i-th layer network, where g(V Hi ,V Li ) is determined using the following formula:

[0027]

[0028] The present invention also provides an energy scheduling device, which is applied to a self-powered visual perception system, and the device includes: a monitoring module, which is used to monitor the energy stored in the energy storage capacitor in real time, wherein the energy stored in the energy storage capacitor is obtained based on the energy adopted by the pixel array; a processing module, which is used to call the pixel array to calculate the first layer network of the image recognition neural network based on the monitoring results, and\or call the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network, wherein the auxiliary layer network is the other layer network of the image recognition neural network except the first layer network; a generation module, which is used to obtain the recognition result of the image recognition neural network based on the calculation result of the pixel array and the calculation result of the in-memory computing unit.

[0029] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an energy scheduling method as described above is implemented.

[0030] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the energy scheduling method as described in any one of the above is implemented.

[0031] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the energy scheduling method as described above is implemented.

[0032] The self-powered visual perception system, energy scheduling method, device and electronic device provided by the present invention, wherein the self-powered visual perception system uses a pixel array to convert received light energy into electrical energy, and then processes the photoelectrically converted energy through a power management unit and stores it on an energy storage capacitor. Under the control of the energy scheduling module, the energy stored in the energy storage capacitor is distributed to the pixel array and the in-memory computing unit in real time, and the calculation of the image recognition neural network is completed layer by layer, thereby achieving in-situ self-powering under the premise of supporting the operation of the image recognition neural network, thereby improving energy utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 This is one of the structural schematic diagrams of the self-powered visual perception system provided by the present invention;

[0035] Figure 2 This is the second structural schematic diagram of the self-powered visual perception system provided by the present invention;

[0036] Figure 3 It is one of the flow charts of the energy scheduling method provided by the present invention;

[0037] Figure 4 It is a schematic diagram of the energy source dispatching principle of the self-powered visual perception system provided by the present invention;

[0038] Figure 5 This is the second flow chart of the energy scheduling method provided by the present invention;

[0039] Figure 6 It is a schematic diagram of a flow chart of determining an execution start threshold value provided by the present invention;

[0040] Figure 7 is a schematic diagram of a flow chart of determining an error tolerance provided by the present invention;

[0041] Figure 8 It is a schematic diagram of a process of introducing a calculation error into each initial layer network of an initial image recognition neural network provided by the present invention;

[0042] Fig. 9 It is a schematic diagram of a flow chart of determining the capacitance value of an energy storage capacitor based on a lower limit power supply voltage provided by the present invention;

[0043] Fig.10 It is a schematic structural diagram of the energy scheduling device provided by the present invention;

[0044] Fig.11 It is a schematic structural diagram of the electronic device provided by the present invention.

[0045] Reference numerals:

[0046] Self-powered visual perception system: 10; Energy storage capacitor: 20;

[0047] Pixel array: 101; In-memory computing unit: 102;

[0048] Power management unit: 103; Energy scheduling module: 104. Detailed implementation manners

[0049] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts fall within the protection scope of the present invention.

[0050] The self-powered visual perception system provided by the present invention can be applied to perform operations on an image recognition neural network to obtain an identification result of the image recognition neural network. In one application scenario, based on the identification result of the image recognition neural network, it can be used to identify a face image to be identified and perform identification processing on the face image to be identified. In one example, the image recognition neural network can be a binary neural network.

[0051] Figure 1 It is one of the schematic structural diagrams of the self-powered visual perception system provided by the present invention.

[0052] The following will be combined with Figure 1 to illustrate the structure of the self-powered visual perception system.

[0053] In an exemplary embodiment of the present invention, in combination with Figure 1 it can be known that the self-powered visual perception system 10 may include a pixel array 101, an in-memory computing unit 102, a power management unit 103, and an energy scheduling module 104. Each module will be introduced separately below.

[0054] It should be noted that Figure 1 the solid arrows shown in Figure 1 represent the energy flow, and

[0055] In one embodiment, the power management unit 103 can be electrically connected to the pixel array 101, and is used to boost the energy collected by the pixel array 101, and store the processed energy in the energy storage capacitor 20. Among them, the pixel array 101 integrates information collection, such as collecting facial image information to be recognized, information processing and energy collection functions. Among them, the information processing function is mainly used to calculate the first layer of the image recognition neural network. It should be noted that while the pixel array 101 completes the calculation of the first layer of the image recognition neural network, it can collect light energy in the environment as an energy source for the operation of the self-powered visual perception system 10.

[0056] The power management unit 103 is used to process the energy collected by the pixel array 101, boost the energy and store it in the energy storage capacitor 20. It should be noted that the energy storage capacitor 20 can be an energy storage capacitor of the self-powered visual perception system 10 itself, or it can be an energy storage capacitor independent of the self-powered visual perception system 10.

[0057] In another embodiment, the energy scheduling module 104 is respectively connected to the energy storage capacitor 20, the pixel array 101 and the in-memory computing unit 102 for real-time monitoring of the energy stored in the energy storage capacitor 20, and based on the monitoring results, the pixel array 101 and / or the in-memory computing unit 102 are called to calculate the image recognition neural network. It can be understood that the energy scheduling module 104 is used to dynamically schedule the energy in the energy storage capacitor 20 to determine the task currently completed by the self-powered visual perception system 10, and then control the working mode of each module of the self-powered visual perception system 10.

[0058] The pixel array 101 and the in-memory computing unit 102 are electrically connected to the energy storage capacitor 20, respectively. The pixel array 101 calculates the first layer network of the image recognition neural network based on the energy stored in the energy storage capacitor 20. The in-memory computing unit 102 calculates the auxiliary layer network of the image recognition neural network based on the energy stored in the energy storage capacitor 20. The auxiliary layer network is the other layer network of the image recognition neural network except the first layer network.

[0059] It can be understood that each layer network in the image recognition neural network is composed of a first layer network and an auxiliary layer network.

[0060] In one embodiment, the image recognition neural network may include a first layer to an Nth layer network, wherein the first layer network may be understood as a first layer network, and the second layer to the Nth layer network may be understood as an auxiliary layer network.

[0061] The self-powered visual perception system provided by the present invention utilizes a pixel array to convert received light energy into electrical energy, and then processes the photoelectrically converted energy through a power management unit and stores it in an energy storage capacitor. Under the control of an energy scheduling module, the energy stored in the energy storage capacitor is distributed to the pixel array and the in-memory computing unit in real time, and the calculation of the image recognition neural network is completed layer by layer, thereby achieving in-situ self-powering while supporting the operation of the image recognition neural network, thereby improving energy utilization.

[0062] Figure 2 This is the second structural schematic diagram of the self-powered visual perception system provided by the present invention.

[0063] The following will be combined Figure 2 The structure of the self-powered visual perception system is explained.

[0064] In another exemplary embodiment of the present invention, Figure 2 It can be seen that the self-powered visual perception system can also include an activation value cache module, wherein the activation value cache module is electrically connected to the pixel array and the in-memory computing unit, respectively, and is used to store the calculation results of the pixel array and the in-memory computing unit. It should be noted that the in-memory computing unit corresponds to the in-memory computing unit #1, in-memory computing unit #2...in-memory computing unit #3 in the figure.

[0065] In yet another exemplary embodiment of the present invention, Figure 2 For explanation, the self-powered visual perception system may also include a read-only memory and a startup module. The startup module is electrically connected to the energy storage capacitor, and is used to start and send a startup signal based on the energy stored in the energy storage capacitor. The read-only memory is communicatively connected to the startup module, and is used to load the weight data of the first layer network of the stored image recognition neural network into the pixel array when the read-only memory receives the startup signal sent by the startup module, so that the pixel array calculates the first layer network based on the weight data of the first layer network, and loads the weight data of the auxiliary layer network of the stored image recognition neural network into the in-memory computing unit, so that the in-memory computing unit calculates the auxiliary layer network based on the weight data of the auxiliary layer network.

[0066] In another exemplary embodiment of the present invention, the in-memory computing unit may include a plurality of in-memory computing sub-units (corresponding to Figure 2 In-memory computing unit #1, in-memory computing unit #2, ..., in-memory computing unit #N). The in-memory computing sub-units correspond to each auxiliary layer network of the image recognition neural network, and perform calculations on each corresponding auxiliary layer network.

[0067] In one embodiment, the startup module realizes autonomous cold start in the startup phase of the self-powered visual perception system and sends a control signal (also called a startup signal). Further, the read-only memory receives the startup signal sent by the startup module and loads the neural network weight data into the pixel array and the in-memory computing unit respectively through the read-only memory.

[0068] During the startup phase of the self-powered visual perception system, the pixel array performs energy collection and charges the energy storage capacitor through the energy management unit. As the voltage of the external energy storage capacitor of the self-powered visual perception system increases and reaches the cold start threshold voltage of the startup module, the startup module turns on and sends a startup signal to turn on the read-only memory. The read-only memory loads the first layer weight data of the image recognition neural network into the pixel array, and loads the second to Nth layer weight data into the corresponding in-memory computing subunit of the in-memory computing subunit (corresponding to the in-memory computing unit #1, in-memory computing unit #2...in-memory computing unit #N in the diagram, respectively). After the self-powered visual perception system is started, it will enter a normal working state and perform self-powered image recognition neural network calculations. It can be understood that by calculating the first layer network based on the pixel array and calculating the auxiliary layer network based on the in-memory computing unit, the image recognition neural network can be calculated.

[0069] In an exemplary embodiment of the present invention, continue to combine Figure 2 To illustrate, the energy scheduling module may include a logic control unit and a plurality of level detection units (corresponding to the level detection unit #0, level detection unit #1 ... level detection unit #N in the diagram, respectively). The level detection units correspond to the pixel array and the in-memory computing subunit, respectively. In one example, the level detection unit #0 corresponds to the pixel array, and the level detection unit #1 ... level detection unit #N corresponds to the in-memory computing subunit #1 ... in-memory computing subunit #N, respectively.

[0070] During the application process, the level detection unit is used to monitor in real time whether the energy stored in the energy storage capacitor meets the execution start threshold, wherein the execution start threshold is the minimum energy value required for the pixel array and the in-memory computing subunit to operate. It should be noted that the pixel array and each in-memory computing subunit respectively correspond to an execution start threshold. When it is detected that the energy of the energy storage capacitor meets the execution start threshold corresponding to the pixel array, the logic control unit can be used to call the pixel array to calculate the first layer network of the image recognition neural network. When it is detected that the energy of the energy storage capacitor meets the execution start threshold corresponding to the in-memory computing subunit, the logic control unit can be used to call the in-memory computing subunit to calculate the auxiliary layer network of the image recognition neural network.

[0071] It should be noted that different in-memory computing sub-units calculate different auxiliary layer networks. Figure 2 To illustrate, for the in-memory computing unit #1, the auxiliary layer network of the second layer can be calculated, for the in-memory computing unit #2, the auxiliary layer network of the third layer can be calculated... For the in-memory computing unit #N, the auxiliary layer network of the N+1th layer can be calculated.

[0072] According to the foregoing description, the self-powered visual perception system provided by the present invention uses a pixel array to convert the received light energy into electrical energy, and then processes the photoelectric energy through a power management unit and stores it on an energy storage capacitor. Under the control of the energy scheduling module, the energy stored in the energy storage capacitor is distributed to the pixel array and the in-memory computing unit in real time, and the calculation of the image recognition neural network is completed layer by layer, thereby achieving in-situ self-powering under the premise of supporting the operation of the image recognition neural network, thereby improving energy utilization.

[0073] The present invention also provides an energy scheduling method, wherein the energy scheduling method is applied to the self-powered visual perception system described above. It can be understood that the energy scheduling method adopts a cyclic working method to distribute the energy stored in the energy storage capacitor to each processing module (corresponding to the pixel array and the in-memory computing unit) layer by layer according to the characteristics of each layer of the image recognition neural network for calculation and processing.

[0074] Figure 3 This is one of the flow charts of the energy scheduling method provided by the present invention.

[0075] The following will be combined Figure 3 The process of the energy scheduling method is explained.

[0076] In an exemplary embodiment of the present invention, Figure 3 It can be seen that the energy scheduling method may include steps 310 to 330, and each step will be introduced below.

[0077] In step 310, energy stored in an energy storage capacitor is monitored in real time, wherein the energy stored in the energy storage capacitor is obtained based on energy used by the pixel array.

[0078] In one embodiment, the energy stored in the energy storage capacitor can be monitored in real time based on the energy scheduling module, and the pixel array or the in-memory computing unit can be called based on the energy stored in the energy storage capacitor to perform calculations on the image recognition neural network. It should be noted that the energy scheduling module can use a periodic cycle method to distribute and schedule the energy stored in the energy storage capacitor. Through this embodiment, the self-powered visual perception system can complete the computing tasks of the image recognition neural network layer by layer with low power consumption and self-power.

[0079] In step 320, based on the monitoring results, the pixel array is called to calculate the first layer network of the image recognition neural network, and\or the in-memory computing unit is called to calculate the auxiliary layer network of the image recognition neural network, wherein the auxiliary layer network is the other layer network of the image recognition neural network except the first layer network.

[0080] In step 330, based on the calculation results of the pixel array and the calculation results of the in-memory computing unit, the recognition results of the image recognition neural network are obtained.

[0081] In one embodiment, based on the monitoring result of the energy stored in the energy storage capacitor monitored in real time by the energy scheduling module, the pixel array can be called to calculate the first layer network of the image recognition neural network, and\or the in-memory computing unit can be called to calculate the auxiliary layer network of the image recognition neural network. Further, the recognition result of the image recognition neural network can be obtained based on the calculation result of the pixel array and the calculation result of the in-memory computing unit.

[0082] In one embodiment, the in-memory computing unit may include a plurality of in-memory computing sub-units, wherein the in-memory computing sub-units correspond to each auxiliary layer network respectively. Further, calling the in-memory computing unit to perform calculations on the auxiliary layer network of the image recognition neural network may be implemented in the following manner:

[0083] The in-memory computing subunit is called to perform calculations on the auxiliary layer network of the image recognition neural network corresponding to the in-memory computing subunit.

[0084] Continue with Figure 2 Taking the illustrated embodiment as an example, the in-memory computing subunit may include the in-memory computing unit #1 ... in-memory computing unit #N shown in the diagram. In the application process, the in-memory computing unit #1 can calculate the auxiliary layer network of the second layer; the in-memory computing unit #2 can calculate the auxiliary layer network of the third layer...; the in-memory computing unit #N can calculate the auxiliary layer network of the N+1th layer. In this embodiment, by calling the in-memory computing subunit to calculate the image recognition neural network layer by layer, the computing power consumption can be reduced, which provides a basis for realizing in-situ self-power supply.

[0085] In another embodiment of the present invention, based on the monitoring results, calling the pixel array to calculate the first layer network of the image recognition neural network, and\or calling the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network can be implemented in the following manner:

[0086] If it is monitored that the energy stored in the energy storage capacitor meets the execution start threshold, the pixel array is called to calculate the first layer network of the image recognition neural network, and\or the in-memory computing unit is called to calculate the auxiliary layer network of the image recognition neural network.

[0087] In one embodiment, the energy values ​​required for calculating different layers of the image recognition neural network are different. When it is monitored that the energy stored in the energy storage capacitor meets the execution start threshold (corresponding to the energy required for calculating different layers of the image recognition neural network), the pixel array can be called to calculate the first layer of the image recognition neural network, and\or the in-memory computing unit can be called to calculate the auxiliary layer of the image recognition neural network.

[0088] Figure 5 This is the second flow chart of the energy scheduling method provided by the present invention.

[0089] The following will be combined Figure 5 The process of the energy scheduling method is explained.

[0090] In another exemplary embodiment of the present invention, Figure 5 It can be seen that the energy scheduling method may include steps 510 to 540, wherein steps 510 to 520 are the same or similar to steps 310 to 320, and step 540 is the same or similar to step 330. For its specific implementation and beneficial effects, please refer to the previous description, which will not be repeated in this embodiment. Step 530 will be introduced below.

[0091] In step 530, if it is monitored that the energy stored in the energy storage capacitor does not meet the execution start threshold, energy can be collected based on the pixel array.

[0092] It is understandable that when the energy stored in the energy storage capacitor does not meet the execution start-up threshold, the working mode of the self-powered visual perception system is in a dormant state. At this time, the pixel array is required to collect external energy to charge the energy storage capacitor to improve the storage capacity of the energy storage capacitor, and lay the foundation for the self-powered visual perception system to perform self-powered operations based on the storage capacity of the energy storage capacitor.

[0093] In another exemplary embodiment of the present invention, the energy values ​​required for calculating different layers of the image recognition neural network are not the same. In one example, the execution start threshold may include a first execution start threshold and a second execution start threshold. The first execution start threshold is the minimum energy value required when the pixel array is running, that is, the first execution start threshold is the minimum energy value required when the pixel array calculates the first layer of the image recognition neural network. The second execution start threshold is the minimum energy value required when the in-memory computing subunit is running, that is, the second execution start threshold is the minimum energy value required when the in-memory computing subunit calculates the auxiliary layer of the image recognition neural network.

[0094] It should be noted that, since there are multiple in-memory computing sub-units and multiple auxiliary layer networks calculated by the in-memory computing sub-units, there are multiple second execution start thresholds and the numerical values ​​of each second execution start threshold may be different.

[0095] Continue with Figure 2 The embodiment shown in the figure is used as an example for explanation. The in-memory computing sub-unit may include the in-memory computing unit #1 ... in-memory computing unit #N shown in the figure. In the application process, the in-memory computing unit #1 can calculate the auxiliary layer network of the second layer. At this time, the second execution start threshold corresponding to the in-memory computing unit #1 can be recorded as E th,2 ; The in-memory computing unit #2 can calculate the auxiliary layer network of the third layer. At this time, the second execution start threshold corresponding to the in-memory computing unit #2 can be recorded as E th,3 ……; the in-memory computing unit #N can calculate the auxiliary layer network of the N+1th layer. At this time, the second execution start threshold corresponding to the in-memory computing unit #N can be recorded as E th,N+1 In addition, the first execution start threshold corresponding to the pixel array can be recorded as E th,1 Among them, E th,1 ……E th,N+1 Can be different from each other.

[0096] Figure 4 It is a schematic diagram of the energy source scheduling principle of the self-powered visual perception system provided by the present invention.

[0097] In order to further explain the energy scheduling method provided by the present invention, Figure 4 The process of the energy source scheduling principle of the self-powered visual perception system is explained.

[0098] In an exemplary embodiment of the present invention, Figure 4In the figure, the working status of each submodule in the self-powered visual perception system is shown with the change over time as the horizontal axis: the first row describes the working mode of the pixel array; the second row shows the energy storage status of the energy storage capacitor; the third row is the working mode of the self-powered visual perception system; the fourth row shows the working status of the in-memory computing unit and the specific in-memory computing subunit number that is activated.

[0099] The energy scheduling module in the self-powered visual perception system adopts a cyclic working method to distribute the energy stored in the energy storage capacitor to each processing module layer by layer according to the characteristics of each layer of the image recognition neural network.

[0100] The specific working process of the self-powered visual perception system is as follows: First, the system is put into sleep mode. In this mode, the pixel array in the system works in energy collection mode, and the power management unit processes the collected energy and stores the energy in the energy storage capacitor, accumulating energy continuously. When the energy accumulated in the energy storage capacitor reaches the execution start threshold E of the first layer of the image recognition neural network, the energy accumulated in the energy storage capacitor reaches the execution start threshold E of the first layer of the image recognition neural network. th,1 , the system switches to active mode. At the same time, the pixel array switches to computing mode to perform the first layer of computing tasks of the image recognition neural network. After the first layer of computing is completed, the system enters sleep mode again, and the pixel array switches back to energy collection mode and remains in this mode until the next round of first layer computing of the image recognition neural network is required. Assuming that there is always light energy in the surrounding environment to meet the continuous energy input to the system, when the system is in sleep mode, the energy stored in the energy storage capacitor will accumulate over time. When the accumulated energy increases to the execution start threshold E of the second layer of the image recognition neural network th,2 , the in-memory computing unit #1 (corresponding to the in-memory computing subunit) will be activated and perform the computing task of the second layer of the image recognition neural network. After completing the computing of the second layer of the image recognition neural network, the system enters sleep mode again. By repeating the above process continuously, the computing process of the entire image recognition neural network can be completed.

[0101] The following will be combined Figure 2 and Figure 4 The content shown explains the scheduling principle of the energy scheduling method.

[0102] At the beginning of the operation of the self-powered visual perception system, only the level detection unit #0 and the logic control unit are turned on. The level detection unit #0 tracks and monitors in real time whether the energy of the energy storage capacitor reaches the execution start threshold of the first layer of the image recognition neural network, thereby realizing the control of the current mode of the system and the execution tasks of each module.

[0103] When the energy collected in the pixel array is sufficient to support the computing task of the first layer of the image recognition neural network, that is, the level detection unit #0 in the energy scheduling module detects that the energy on the energy storage capacitor outside the system reaches the execution start threshold E of the first layer of the network th,1 When the energy is on, the logic control unit in the energy scheduling module sends a control signal to drive the activation value cache module to start and control the pixel array to switch from the energy collection mode to the calculation mode, the system working mode switches from the sleep mode to the activation mode, and the level detection unit #0 is turned off.

[0104] Furthermore, the pixel array completes the calculation task of the first layer of the image recognition neural network on the analog circuit, and stores the calculation results of the first layer in the activation value cache module, thereby eliminating the need for data conversion and storage of the original image.

[0105] After the first layer of calculation is completed, the logic control unit in the energy scheduling module sends a control signal to close the activation value cache module, and at the same time controls the pixel array to switch from the calculation mode to the energy collection mode, the system working mode switches from the activation mode to the sleep mode, and the level detection unit #1 is turned on. After that, the pixel array always remains in the energy collection mode before the next round of image recognition neural network first layer calculation is required.

[0106] Due to the first layer of calculation, the energy stored in the energy storage capacitor gradually decreases and is insufficient to support the second layer of calculation tasks. Until the pixel array switches from the calculation mode back to the energy collection mode, the energy stored in the energy storage capacitor gradually increases. Therefore, the level detection unit #1 continuously detects whether the energy on the energy storage capacitor reaches the execution start threshold E of the second layer of the network. th,2 Once it is detected that the energy on the energy storage capacitor reaches the execution start threshold E of the second layer of the network, th,2 , the logic control unit in the energy scheduling module sends a control signal to drive the activation value cache module to start and control the in-memory calculation unit #1 to start, the system working mode is switched from sleep mode to activation mode, and the level detection unit #2 is turned off. The activation value cache module inputs the first-layer calculation result into the in-memory calculation unit #1, and starts the second-layer calculation in the in-memory calculation unit #1. After the calculation is completed, the in-memory calculation unit #1 stores the second-layer output result in the activation value cache module, and the logic control unit in the energy scheduling module sends a control signal to turn off the activation value cache module and the in-memory calculation unit #1, the system working mode is switched from activation mode to sleep mode, and the level detection unit #3 is turned on. Thereafter, before the next round of calculation of the second layer of the image recognition neural network is required, the in-memory calculation unit #1 always remains in a closed state.

[0107] Based on the same principle, all subsequent calculations are completed in the corresponding in-memory calculation subunits until the calculation of the last layer is completed. Assuming that the image recognition neural network has N+1 layers, the system calculates the last layer as follows: the level detection unit #N continuously detects whether the energy stored in the energy storage capacitor reaches the execution start threshold E of the network N+1 layer. th,N+1 Once the energy on the energy storage capacitor reaches the execution start threshold E th,N+1 , the logic control unit in the energy scheduling module sends a control signal to drive the activation value cache module to start and control the in-memory calculation unit #N to start. The system working mode is switched from sleep mode to activation mode, and the level detection unit #N is turned off. The activation value cache module inputs the calculation result of the Nth layer of the network into the in-memory calculation unit #N, and starts the N+1th layer calculation in the in-memory calculation unit #N. After the calculation is completed, the in-memory calculation unit #N stores the N+1th layer result in the activation value cache module, and the logic control unit in the energy scheduling module sends a control signal to turn off the activation value cache module and the in-memory calculation unit #N. The system working mode is switched from activation mode to sleep mode, and the level detection unit #0 is turned on.

[0108] It should be noted that when the image recognition neural network needs to be calculated again, it is possible to return to the process of preparing the first layer of calculations.

[0109] It is understandable that, for the face recognition scenario, performing a face recognition operation requires completing a calculation of the image recognition neural network.

[0110] In this embodiment, the energy collected in situ by the pixel array can be distributed for low-power layer-by-layer calculation of the image recognition neural network. Thus, the energy collected by the pixel array can be fully utilized, the power consumption of completing the image recognition neural network calculation can be reduced, and the calculation accuracy can be guaranteed. When completing the image recognition neural network calculation of each layer, repeated calculations and data loss due to energy interruption or insufficient energy will not occur. Further, when the energy stored in the energy storage capacitor outside the self-powered visual perception system is insufficient to support the accurate completion of the image recognition neural network layer that needs to be calculated at present, that is, when the voltage of the energy storage capacitor outside the system does not reach the execution start threshold voltage, the system is in sleep mode, the pixel array continuously collects energy, and stores the energy in the energy storage capacitor outside the system after being processed by the power management unit. When the voltage of the energy storage capacitor outside the system reaches the execution start threshold voltage of the image recognition neural network layer that needs to be calculated at present, the system switches to the activation mode, and the pixel array or the corresponding in-memory computing unit completes the calculation of the image recognition neural network layer.

[0111] Figure 6 It is a schematic diagram of a flow chart of determining an execution start threshold value provided by the present invention.

[0112] The following will be combined Figure 6 The process of determining the execution start threshold is described.

[0113] In an exemplary embodiment of the present invention, Figure 6 It can be known that determining the execution start threshold may include steps 610 to 640, and each step will be described below.

[0114] In step 610, the error tolerance of each layer of the image recognition neural network is determined respectively.

[0115] In one embodiment, if the image recognition neural network includes N layers of networks, the error tolerance of each layer of the network can be determined separately.

[0116] Figure 7 This is a schematic diagram of the process of determining the error tolerance provided by the present invention. Figure 7 The process of determining the error tolerance of each layer of the network is explained.

[0117] In one embodiment, in combination Figure 7 It can be seen that determining the error tolerance may include steps 710 to 730, and each step will be introduced below.

[0118] In step 710, an initial image recognition neural network is determined.

[0119] In one embodiment, the initial image recognition neural network is obtained based on a pre-training method. It is understandable that the initial image recognition neural network can obtain an image recognition neural network after retraining. When the reasoning accuracy of the retrained initial image recognition neural network reaches the accuracy threshold, it can be considered that the initial image recognition neural network has obtained an image recognition neural network after retraining.

[0120] In step 720, a computational error is introduced into each initial layer of the initial image recognition neural network, wherein the computational error is obtained by binary search, so that the inference accuracy of the initial image recognition neural network with the computational error inserted reaches an accuracy threshold after layer-by-layer retraining, wherein the accuracy threshold is determined according to the image recognition neural network.

[0121] In step 730, the error value of the calculated error is used as the error tolerance.

[0122] In one embodiment, a computational error is introduced into each layer of the initial image recognition neural network and the initial image recognition neural network is retrained. When the inference accuracy after retraining reaches an accuracy threshold, the error value of the computational error introduced into each layer can be defined as the error tolerance of the current layer.

[0123] It should be noted that the input of the layer-by-layer retraining method is the pre-trained model of the image recognition neural network (corresponding to the initial image recognition neural network) and an acceptable accuracy threshold, and the output is the error value of the computational error that each layer of the initial image recognition neural network can tolerate.

[0124] In the application process, a network layer that has not been injected with calculation errors is selected at each step. By introducing calculation errors into it, the maximum calculation error rate that the layer can tolerate when the accuracy threshold is met (corresponding to the maximum error rate that can be tolerated) is found, and this error rate is defined as the error tolerance of the layer. Then the initial image recognition neural network is retrained to restore a certain inference accuracy, and the retrained model is used as the initial model for the next step. Whenever the error tolerance of a layer is determined, the corresponding error amount will be injected into the calculation results of the layer in all subsequent steps.

[0125] Different layers of the initial image recognition neural network have different characteristics: on the one hand, different layers have different error tolerance capabilities when introducing the same error; on the other hand, the energy consumption of each layer of the initial image recognition neural network is proportional to the number of operations in the current layer. Therefore, it is necessary to jointly consider the layered error tolerance elasticity (corresponding to the single-layer error tolerance value) and the amount of calculation (corresponding to the single-layer calculation amount).

[0126] Figure 8 This is a flow chart of introducing calculation errors into each initial layer network of the initial image recognition neural network provided by the present invention. Figure 8 The process of introducing computational errors into each initial layer of the initial image recognition neural network is explained.

[0127] In an exemplary embodiment of the present invention, introducing a computational error into each initial layer network of the initial image recognition neural network may include steps 810 to 840, and each step will be described below.

[0128] In step 810, based on the initial image recognition neural network, the single-layer error tolerance value and the single-layer calculation amount of each initial layer network of the initial image recognition neural network are determined, wherein the single-layer error tolerance value is the maximum error rate that the initial layer network can bear when the inference accuracy reaches the accuracy threshold under the condition of introducing the calculation error. It should be noted that the calculation error is the calculation error introduced into a certain layer in the initial image recognition neural network.

[0129] In step 820, based on the single-layer error tolerance value and the single-layer calculation amount, a retraining factor of each initial layer network of the initial image recognition neural network is calculated.

[0130] In the layer-by-layer retraining method, the retraining factor RTF is used to determine the error injection order of the network layer. The initial image recognition neural network layer with a higher RTF will be injected with calculation errors earlier and retrained. Each time error injection and retraining is performed, the calculation accuracy of the initial model of the next step will be lower than the calculation accuracy of the initial model of the current step. Therefore, the initial image recognition neural network layer with a higher RTF can be injected with a larger calculation error. The fact that a certain layer of the network can be injected with more errors means that the operating voltage of the computing circuit of this layer can be lower, and thus the energy consumption of a single calculation is also lower. Therefore, in order to reduce the energy consumption of image recognition neural network calculations, neural network layers with higher error recovery capabilities for calculation errors and neural network layers with more calculations should have higher RTFs.

[0131] In one example, the retraining factor RTF calculation formula of each initial layer network of the initial image recognition neural network can be defined as formula (1):

[0132] RTF=ERR norm ×OP norm (1)

[0133] Among them, ERR norm Represents the single layer error tolerance value, OP norm Represents the amount of calculation for a single layer. Among them, the single layer error tolerance value ERR norm It can be understood as the error tolerance value of the current layer normalized relative to the maximum error tolerance value of all layers, that is, the error tolerance value of a single layer refers to the maximum error rate that a layer can withstand to ensure the inference accuracy threshold when only the calculation error is introduced into a certain initial layer network. norm It is the amount of computation of the current layer normalized relative to the maximum amount of computation in all layers.

[0134] In step 830, the retraining factors are sorted in descending order to obtain an error injection order.

[0135] In step 840, a computational error is introduced into each initial layer network of the initial image recognition neural network according to the error injection sequence.

[0136] In one embodiment, the retraining factors corresponding to each layer (corresponding to the initial layer network) of the initial image recognition neural network can be sorted from large to small to obtain an error injection sequence. Further, according to the error injection sequence, a calculation error is introduced into each initial layer network of the initial image recognition neural network.

[0137] In this embodiment, the layer-by-layer retraining method for the initial image recognition neural network will fully release the tolerance of the calculation error of each layer, so that the network calculation can work within a lower voltage range. Its advantages are: on the one hand, low voltage operation not only relaxes the requirements for capacitor size, but also helps to reduce the energy consumption used in calculation, thereby completing the self-powered image recognition neural network calculation process with a smaller delay; on the other hand, for the neural network layers with more calculation times, a larger calculation error can be introduced, so that these layers can be calculated at a lower voltage, further reducing the energy consumption of the entire calculation process.

[0138] In step 620, based on the error tolerance, a lower limit power supply voltage of each layer of the network is determined, wherein the lower limit power supply voltage corresponds to the error tolerance.

[0139] In one embodiment, the error tolerance can be mapped to the lower limit power supply voltage V of each layer of the image recognition neural network. Li , i=1,2,…,N.

[0140] In this embodiment, the layer-by-layer error tolerance of the image recognition neural network can be directly mapped to the lower limit power supply voltage of each layer of the computing circuit, that is, in order to ensure that the calculation accuracy threshold is reached, the power supply voltage corresponding to the maximum error rate that the layer can withstand is introduced.

[0141] In step 630 , the capacitance value of the energy storage capacitor is determined based on the lower limit power supply voltage.

[0142] Fig. 9 The present invention provides a flow chart of determining the capacitance value of the energy storage capacitor based on the lower limit power supply voltage. Fig. 9 The process of determining the capacitance value of the energy storage capacitor based on the lower limit power supply voltage is described.

[0143] In an exemplary embodiment of the present invention, determining the capacitance value of the energy storage capacitor based on the lower limit power supply voltage may include steps 910 to 930, and each step will be described below.

[0144] In step 910, the standard operating voltage of each layer of the image recognition neural network is determined.

[0145] In step 920 , a supply capacitance value required for the layer network is determined based on a lower limit power supply voltage corresponding to the layer network and a standard operating voltage.

[0146] It should be noted that, based on the determination of the lower limit power supply voltage, the design of the turn-on threshold voltage and the size of the energy storage capacitor depends on the energy consumption characteristics of the computing circuits of each layer of the image recognition neural network. According to the formula (2) of energy and capacitance:

[0147]

[0148] Among them, E C Represents the energy stored in the energy storage capacitor, V C represents the voltage on the energy storage capacitor, C ST Indicates the capacitance value of the energy storage capacitor.

[0149] In one example, if the execution turn-on threshold voltage of all layers is V max , which is also recorded as the standard operating voltage of the computing circuit, the energy storage capacitor required for each layer of calculation (the supply capacitor value required for the corresponding layer network) can be expressed as formula (3):

[0150]

[0151] Among them, C ST,i V represents the supply capacitance value required for the i-th layer network in the image recognition neural network; max Indicates standard operating voltage; V Li represents the lower limit power supply voltage corresponding to the i-th layer network; E f,i (g(V max ,V Li )) indicates that at a fixed working voltage g(V max ,V Li ) to calculate the energy consumption value required by the i-th layer network, where g(V max ,V Li ) is determined using the following formula (4):

[0152]

[0153] In step 930 , the maximum supply capacitance value among the supply capacitance values ​​is used as the capacitance value of the energy storage capacitor.

[0154] In one embodiment, the largest supply capacitance value among the supply capacitance values ​​may be used as the capacitance value of the energy storage capacitor. That is, the capacitance value C of the energy storage capacitor ST It can be determined using the following formula (5):

[0155] C ST =max{C ST,i |i=1,2,…,N} (5)

[0156] Among them, i represents the number of layers of the image recognition neural network.

[0157] In step 640, an execution start threshold corresponding to the layer network is determined based on the capacitance value of the energy storage capacitor and the lower limit power supply voltage corresponding to the layer network.

[0158] In one embodiment, based on the capacitance value of the energy storage capacitor and the lower limit power supply voltage corresponding to the layer network, determining the execution start threshold corresponding to the layer network can be implemented by using the following formula (6):

[0159]

[0160] Among them, V Hi represents the execution start threshold corresponding to the i-th layer network; C ST Indicates the capacitance value of the energy storage capacitor; V Li represents the lower limit power supply voltage corresponding to the i-th layer network; E f,i (g(V Hi ,V Li )) indicates that at a fixed working voltage g(V Hi ,V Li ) to calculate the energy consumption value required by the i-th layer network, where g(V Hi ,V Li ) can be determined using the following formula (7):

[0161]

[0162] It should be noted that formula (6) is a nonlinear equation in the form of x=f(x), which can be solved by iteration. Hi The initial value is chosen to be V max .

[0163] According to the above description, the energy scheduling method provided by the present invention can allocate the energy collected in situ by the pixel array for low-power layer-by-layer calculation of the image recognition neural network. In this way, the energy collected by the pixel array can be fully utilized, the power consumption of completing the image recognition neural network calculation can be reduced, and the calculation accuracy can be guaranteed. When completing the image recognition neural network calculation of each layer, there will be no repeated calculation and data loss due to energy interruption or insufficient energy. And by determining the execution start threshold, high-efficiency self-powered image recognition neural network calculation can be achieved.

[0164] Based on the same concept, the present invention also provides an energy scheduling device.

[0165] The energy scheduling device provided by the present invention is described below. The energy scheduling device described below and the energy scheduling method described above can be referenced to each other.

[0166] Fig.10 It is a structural schematic diagram of the energy scheduling device provided by the present invention.

[0167] In an exemplary embodiment of the present invention, the energy scheduling device can be applied to the self-powered visual perception system described above. Fig.10It can be seen that the energy scheduling device may include a monitoring module 1010, a processing module 1020 and a generating module 1030, and each module will be introduced below.

[0168] The monitoring module 1010 may be configured to monitor the energy stored in the energy storage capacitor in real time, wherein the energy stored in the energy storage capacitor is obtained based on the energy used by the pixel array.

[0169] The processing module 1020 can be configured to call the pixel array to perform calculations on the first layer network of the image recognition neural network based on the monitoring results, and\or call the in-memory computing unit to perform calculations on the auxiliary layer network of the image recognition neural network, wherein the auxiliary layer network is other layer networks of the image recognition neural network except the first layer network.

[0170] The generation module 1030 may be configured to obtain a recognition result of the image recognition neural network based on the calculation result of the pixel array and the calculation result of the in-memory computing unit.

[0171] In an exemplary embodiment of the present invention, the in-memory computing unit may include multiple in-memory computing sub-units, wherein the in-memory computing sub-units correspond to each auxiliary layer network respectively; the processing module 1020 may call the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network in the following manner: call the in-memory computing sub-unit to calculate the auxiliary layer network of the image recognition neural network corresponding to the in-memory computing sub-unit.

[0172] In an exemplary embodiment of the present invention, the processing module 1020 can use the following method to call the pixel array to calculate the first layer network of the image recognition neural network based on the monitoring results, and\or call the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network: if it is monitored that the energy stored in the energy storage capacitor meets the execution start-up threshold, then call the pixel array to calculate the first layer network of the image recognition neural network, and\or call the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network.

[0173] In an exemplary embodiment of the present invention, the execution start threshold may include a first execution start threshold and a second execution start threshold; the processing module 1020 may be implemented in the following manner: if it is monitored that the energy stored in the energy storage capacitor meets the execution start threshold, the pixel array is called to calculate the first layer network of the image recognition neural network, and\or the in-memory calculation unit is called to calculate the auxiliary layer network of the image recognition neural network: if it is monitored that the energy stored in the energy storage capacitor meets the first execution start threshold, the pixel array is called to calculate the first layer network of the image recognition neural network, wherein the first execution start threshold is the minimum energy value required for the pixel array to operate; if it is monitored that the energy stored in the energy storage capacitor meets the second execution start threshold, the in-memory calculation subunit is called to calculate the auxiliary layer network of the image recognition neural network, wherein the second execution start threshold is the minimum energy value required for the in-memory calculation subunit to operate.

[0174] In an exemplary embodiment of the present invention, the processing module 1020 may also be configured to collect energy based on the pixel array if it is monitored that the energy stored in the energy storage capacitor does not meet the execution start threshold.

[0175] In an exemplary embodiment of the present invention, the processing module 1020 can determine the execution start threshold in the following manner: determine the error tolerance of each layer of the network in the image recognition neural network respectively; based on the error tolerance, determine the lower limit power supply voltage of each layer of the network, wherein the lower limit power supply voltage corresponds to the error tolerance; based on the lower limit power supply voltage, determine the capacitance value of the energy storage capacitor; based on the capacitance value of the energy storage capacitor and the lower limit power supply voltage corresponding to the layer network, determine the execution start threshold corresponding to the layer network.

[0176] In an exemplary embodiment of the present invention, the processing module 1020 can determine the error tolerance in the following manner: determine the initial image recognition neural network; introduce a computational error into each initial layer network of the initial image recognition neural network, wherein the computational error is obtained by binary search, so that the inference accuracy of the initial image recognition neural network with the computational error inserted reaches an accuracy threshold after layer-by-layer retraining, wherein the accuracy threshold is determined based on the image recognition neural network; and use the error value of the computational error as the error tolerance.

[0177] In an exemplary embodiment of the present invention, the processing module 1020 can introduce a computational error to each initial layer network of the initial image recognition neural network in the following manner: based on the initial image recognition neural network, determine the single-layer error tolerance value and the single-layer calculation amount of each initial layer network of the initial image recognition neural network, wherein the single-layer error tolerance value is the maximum error rate that the initial layer network can bear when the inference accuracy reaches the accuracy threshold when the computational error is introduced; based on the single-layer error tolerance value and the single-layer calculation amount, calculate the retraining factor of each initial layer network of the initial image recognition neural network; sort the retraining factors in descending order to obtain the error injection order; and introduce a computational error to each initial layer network of the initial image recognition neural network according to the error injection order.

[0178] In an exemplary embodiment of the present invention, the processing module 1020 can determine the capacitance value of the energy storage capacitor based on the lower limit power supply voltage in the following manner: determine the standard operating voltage of each layer of the network in the image recognition neural network; determine the supply capacitance value required for the layer network based on the lower limit power supply voltage corresponding to the layer network and the standard operating voltage; and use the largest supply capacitance value among the supply capacitance values ​​as the capacitance value of the energy storage capacitor.

[0179] In an exemplary embodiment of the present invention, the processing module 1020 may use the following formula (8) to determine the supply capacitance value required by the layer network based on the lower limit power supply voltage corresponding to the layer network and the standard operating voltage:

[0180]

[0181] Among them, C ST,i V represents the supply capacitance value required for the i-th layer network in the image recognition neural network; max Indicates standard operating voltage; V Li represents the lower limit power supply voltage corresponding to the i-th layer network; E f,i (g(V max ,V Li )) indicates that at a fixed working voltage g(V max ,V Li ) to calculate the energy consumption value required by the i-th layer network, where g(V max ,V Li ) is determined using the following formula (9):

[0182]

[0183] In an exemplary embodiment of the present invention, the processing module 1020 may use the following formula (10) to determine the execution start threshold corresponding to the layer network based on the capacitance value of the energy storage capacitor and the lower limit power supply voltage corresponding to the layer network:

[0184]

[0185] Among them, V Hi represents the execution start threshold corresponding to the i-th layer network; C ST Indicates the capacitance value of the energy storage capacitor; V Li represents the lower limit power supply voltage corresponding to the i-th layer network; E f,i (g(V Hi ,V Li )) indicates that at a fixed working voltage g(V Hi ,V Li ) to calculate the energy consumption value required by the i-th layer network, where g(V Hi ,V Li ) is determined using the following formula (11):

[0186]

[0187] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 complete mutual communication through the communication bus 1140. The processor 1110 may call the logic instructions in the memory 1130 to execute the energy scheduling method, which is applied to the self-powered visual perception system, and the method includes: real-time monitoring of the energy stored in the energy storage capacitor, wherein the energy stored in the energy storage capacitor is obtained based on the energy used by the pixel array; based on the monitoring result, calling the pixel array to calculate the first layer network of the image recognition neural network, and\or calling the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network, wherein the auxiliary layer network is the other layer network of the image recognition neural network except the first layer network; based on the calculation result of the pixel array and the calculation result of the in-memory computing unit, the recognition result of the image recognition neural network is obtained.

[0188] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0189] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the energy scheduling method provided by the above methods. The energy scheduling method is applied to a self-powered visual perception system, and the method includes: real-time monitoring of the energy stored in the energy storage capacitor, wherein the energy stored in the energy storage capacitor is obtained based on the energy adopted by the pixel array; based on the monitoring results, calling the pixel array to calculate the first layer network of the image recognition neural network, and\or calling the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network, wherein the auxiliary layer network is the other layer network of the image recognition neural network except the first layer network; based on the calculation results of the pixel array and the calculation results of the in-memory computing unit, the recognition result of the image recognition neural network is obtained.

[0190] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the energy scheduling method provided by the above-mentioned methods, and the energy scheduling method is applied to a self-powered visual perception system, the method comprising: real-time monitoring of the energy stored in an energy storage capacitor, wherein the energy stored in the energy storage capacitor is obtained based on the energy adopted by a pixel array; based on the monitoring results, calling the pixel array to calculate the first layer network of the image recognition neural network, and\or calling an in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network, wherein the auxiliary layer network is the other layer network of the image recognition neural network except the first layer network; based on the calculation results of the pixel array and the calculation results of the in-memory computing unit, obtaining the recognition result of the image recognition neural network.

[0191] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative work.

[0192] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0193] It can be further understood that although the operations are described in a specific order in the drawings in the embodiments of the present invention, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring all the operations shown to obtain the desired result. In a specific environment, multitasking and parallel processing may be beneficial.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A self-powered visual perception system, It is characterized in that The self-powered visual perception system is used to operate an image recognition neural network. The system includes: a pixel array, an in-memory computing unit, a power management unit, and an energy scheduling module, wherein: The power management unit is electrically connected to the pixel array, and is used to boost the energy collected by the pixel array and store the processed energy in an energy storage capacitor; The energy scheduling module is respectively connected to the energy storage capacitor, the pixel array and the in-memory computing unit for real-time monitoring of the energy stored in the energy storage capacitor, and based on the monitoring result, calling the pixel array and / or the in-memory computing unit to calculate the image recognition neural network; Wherein, the pixel array and the in-memory computing unit are electrically connected to the energy storage capacitor respectively, the pixel array calculates the first layer network of the image recognition neural network based on the energy stored in the energy storage capacitor, and the in-memory computing unit calculates the auxiliary layer network of the image recognition neural network based on the energy stored in the energy storage capacitor, wherein the auxiliary layer network is other layer networks of the image recognition neural network except the first layer network, wherein the self-powered visual perception system also includes a read-only memory and a startup module, wherein, The starting module is electrically connected to the energy storage capacitor, and is used to start based on the energy stored in the energy storage capacitor and send a starting signal; The read-only memory is communicatively connected to the startup module, and is used for loading the stored weight data of the first layer network of the image recognition neural network into the pixel array when the read-only memory receives the startup signal sent by the startup module, so that the pixel array calculates the first layer network based on the weight data of the first layer network, and loading the stored weight data of the auxiliary layer network of the image recognition neural network into the in-memory computing unit, so that the in-memory computing unit calculates the auxiliary layer network based on the weight data of the auxiliary layer network.

2. The self-powered visual perception system according to claim 1, It is characterized in that The self-powered visual perception system further includes an activation value cache module, wherein: The activation value cache module is electrically connected to the pixel array and the in-memory computing unit respectively, and is used to store the calculation results of the pixel array and the in-memory computing unit.

3. The self-powered visual perception system according to claim 1, It is characterized in that The in-memory computing unit includes a plurality of in-memory computing sub-units, wherein the in-memory computing sub-units respectively correspond to each auxiliary layer network of the image recognition neural network and perform calculations on each corresponding auxiliary layer network.

4. The self-powered visual perception system according to claim 3, It is characterized in that The energy scheduling module includes a logic control unit and a plurality of level detection units, the level detection units respectively corresponding to the pixel array and the in-memory calculation subunit, wherein: The level detection unit is used to monitor in real time whether the energy stored in the energy storage capacitor meets the execution start threshold, wherein the execution start threshold is the minimum energy value required for the pixel array and the in-memory calculation subunit to operate; The logic control unit is used to call the pixel array to calculate the first layer network of the image recognition neural network, and\or call the in-memory computing subunit to calculate the auxiliary layer network of the image recognition neural network.

5. An energy scheduling method, It is characterized in that The energy scheduling method is applied to the self-powered visual perception system according to any one of claims 1 to 4, and the method comprises: monitoring in real time the energy stored in the energy storage capacitor, wherein the energy stored in the energy storage capacitor is obtained based on the energy used by the pixel array; Based on the monitoring results, the pixel array is called to calculate the first layer network of the image recognition neural network, and\or the in-memory computing unit is called to calculate the auxiliary layer network of the image recognition neural network, wherein the auxiliary layer network is the other layer network of the image recognition neural network except the first layer network; Based on the calculation results of the pixel array and the calculation results of the in-memory computing unit, the recognition results of the image recognition neural network are obtained.

6. The energy scheduling method according to claim 5, It is characterized in that The in-memory computing unit includes a plurality of in-memory computing sub-units, wherein the in-memory computing sub-units correspond to each auxiliary layer network respectively; The in-memory computing unit is called to compute the auxiliary layer network of the image recognition neural network, specifically including: The in-memory computing subunit is called to perform calculations on the auxiliary layer network of the image recognition neural network corresponding to the in-memory computing subunit.

7. The energy scheduling method according to claim 5, It is characterized in that Based on the monitoring results, calling the pixel array to calculate the first layer network of the image recognition neural network, and / or calling the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network, specifically includes: If it is monitored that the energy stored in the energy storage capacitor meets the execution start threshold, the pixel array is called to calculate the first layer network of the image recognition neural network, and\or the in-memory computing unit is called to calculate the auxiliary layer network of the image recognition neural network.

8. The energy scheduling method according to claim 7, It is characterized in that The execution start threshold includes a first execution start threshold and a second execution start threshold; If it is monitored that the energy stored in the energy storage capacitor meets the execution start threshold, the pixel array is called to calculate the first layer network of the image recognition neural network, and\or the in-memory computing unit is called to calculate the auxiliary layer network of the image recognition neural network, specifically including: If it is monitored that the energy stored in the energy storage capacitor meets the first execution start threshold, calling the pixel array to calculate the first layer network of the image recognition neural network, wherein the first execution start threshold is the minimum energy value required for the pixel array to operate; If it is monitored that the energy stored in the energy storage capacitor meets the second execution start threshold, the in-memory computing subunit is called to calculate the auxiliary layer network of the image recognition neural network, wherein the second execution start threshold is the minimum energy value required for the in-memory computing subunit to run.

9. The energy scheduling method according to claim 7, It is characterized in that Before calling the pixel array to calculate the first layer network of the image recognition neural network based on the monitoring result, and / or calling the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network, the method further includes: If it is monitored that the energy stored in the energy storage capacitor does not meet the execution start threshold, energy is collected based on the pixel array.

10. The energy scheduling method according to claim 7, It is characterized in that The execution start threshold is determined in the following manner: Determining the error tolerance of each layer of the image recognition neural network respectively; Based on the error tolerance, determining a lower limit power supply voltage of each layer network, wherein the lower limit power supply voltage corresponds to the error tolerance; Based on the lower limit power supply voltage, determining the capacitance value of the energy storage capacitor; The execution start threshold corresponding to the layer network is determined based on the capacitance value of the energy storage capacitor and the lower limit power supply voltage corresponding to the layer network.

11. The energy scheduling method according to claim 10, It is characterized in that The error tolerance is determined in the following manner: Determine the initial image recognition neural network; Introducing a calculation error into each initial layer network of the initial image recognition neural network, wherein the calculation error is obtained by binary search, so that the inference accuracy of the initial image recognition neural network into which the calculation error is inserted reaches an accuracy threshold after layer-by-layer retraining, wherein the accuracy threshold is determined according to the image recognition neural network; The error value of the calculated error is used as the error tolerance.

12. The energy scheduling method according to claim 11, It is characterized in that The step of introducing a computational error into each initial layer network of the initial image recognition neural network specifically includes: Based on the initial image recognition neural network, determining a single-layer error tolerance value and a single-layer calculation amount of each initial layer network of the initial image recognition neural network, wherein the single-layer error tolerance value is a maximum error rate that the initial layer network can bear when the inference accuracy reaches the accuracy threshold under the condition of introducing the calculation error; Calculate the retraining factor of each initial layer network of the initial image recognition neural network based on the single-layer error tolerance value and the single-layer calculation amount; The retraining factors are sorted in descending order to obtain an error injection order; According to the error injection sequence, computational errors are introduced into each initial layer network of the initial image recognition neural network.

13. The energy scheduling method according to claim 10, It is characterized in that The determining the capacitance value of the energy storage capacitor based on the lower limit power supply voltage specifically includes: Determine the standard operating voltage of each layer of the image recognition neural network; Determining a supply capacitance value required for the layer network based on the lower limit power supply voltage corresponding to the layer network and the standard operating voltage; The largest supply capacitance value among the supply capacitance values ​​is used as the capacitance value of the energy storage capacitor.

14. The energy scheduling method according to claim 13, It is characterized in that The supply capacitance value required by the layer network is determined based on the lower limit power supply voltage corresponding to the layer network and the standard operating voltage, and is implemented by using the following formula: ； in, represents the supply capacitance value required for the i-th layer network in the image recognition neural network; Indicates the standard operating voltage; represents the lower limit power supply voltage corresponding to the i-th layer network; Indicates that the fixed working voltage The energy consumption value required for the i-th layer network is calculated as follows: Determined using the following formula: 。 15. The energy scheduling method according to claim 14, It is characterized in that The determination of the execution start threshold corresponding to the layer network based on the capacitance value of the energy storage capacitor and the lower limit power supply voltage corresponding to the layer network is implemented by the following formula: ； in, represents the execution start threshold corresponding to the i-th layer network; represents the capacitance value of the energy storage capacitor; represents the lower limit power supply voltage corresponding to the i-th layer network; Indicates that the fixed working voltage The energy consumption value required for the i-th layer network is calculated as follows: Determined using the following formula: 。 16. An energy dispatching device, It is characterized in that The energy scheduling device is applied to the self-powered visual perception system according to any one of claims 1 to 4, and the device comprises: A monitoring module, used for real-time monitoring of energy stored in the energy storage capacitor, wherein the energy stored in the energy storage capacitor is obtained based on energy used by the pixel array; A processing module, used for invoking the pixel array to calculate the first layer network of the image recognition neural network based on the monitoring result, and / or invoking the in-memory computing unit to calculate the auxiliary layer network of the image recognition neural network, wherein the auxiliary layer network is the other layer network of the image recognition neural network except the first layer network; A generation module is used to obtain the recognition result of the image recognition neural network based on the calculation result of the pixel array and the calculation result of the in-memory calculation unit.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the energy scheduling method as described in any one of claims 5 to 15 is implemented.

18. A non-transitory computer readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the energy scheduling method according to any one of claims 5 to 15 is implemented.

19. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the energy scheduling method according to any one of claims 5 to 15 is implemented.

Images