Quasi-e-system neural network model, construction method, data processing method and electronic equipment
By constructing a quasi-e-ary neural network model and using a combination of binary and ternary functions to represent neurons, the problem of deploying neural networks on hardware-constrained devices is solved, achieving efficient computation and storage optimization while maintaining model accuracy.
Patent Information
- Application Number
- CN202411164691.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2026-03-03
AI Technical Summary
Neural network models, due to their complex structure and large parameter scale, consume huge amounts of computing resources and have high storage requirements, making them difficult to deploy on hardware-constrained devices. While quantization methods reduce computational complexity, they lead to a decrease in accuracy.
A quasi-e-ary neural network model is constructed, which uses a combination of binary and ternary functions to represent neurons. By linearly combining these functions, the full-precision activation function is approximated, reducing storage space and computational complexity while maintaining model accuracy.
While reducing computational complexity and storage requirements, it effectively maintains model accuracy and improves the efficiency of neural networks, making it suitable for resource-constrained embedded and edge devices.
Smart Images
Figure CN121599006A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a quasi-e-ary neural network model and construction method, a data processing method, and an electronic device. Background Technology
[0002] With the rapid development of artificial intelligence technology, neural networks, as a key technology for achieving intelligent tasks, are gradually being applied in various fields. However, due to their complex structure and large parameter scale, neural networks consume enormous computing resources and have extremely high model storage requirements. On the one hand, this limits their application in larger datasets and more complex tasks to some extent. On the other hand, the training and inference of neural networks require high-performance operators and hardware support, such as GPUs, which places high demands on hardware devices. This means that storing and loading neural network models on hardware-constrained electronic devices such as embedded devices and edge devices requires a large amount of storage space and network bandwidth, increasing the cost and maintenance complexity of the devices.
[0003] Therefore, quantization methods for neural networks were proposed and popularized. These methods convert the weights and activation values of the neural network from high-precision floating-point numbers to low-precision numerical values, reducing the model's storage requirements and computational complexity. However, this is a post-processing method, meaning quantization is performed on an already trained model. While this method reduces model size and computational complexity, it leads to a decrease in accuracy. Therefore, a pre-processing method called "Binary Neural Networks (BNNs)" has become a research hotspot—directly constructing a new model instead of quantizing the original model. The weights and activation values of BNNs are strictly limited to the binary range of -1 and 1. After compression using binarization, the weights and activations in the network can be represented by a single bit, without consuming too much memory. Furthermore, through binarization, BNNs can use lightweight XNOR and Bitcount operations instead of cumbersome floating-point multiplication and addition operations. Therefore, it can significantly reduce memory usage, improve execution speed, and reduce energy consumption. Currently, BNNs greatly simplify the computation process, but this has led to a significant decline in model accuracy. Summary of the Invention
[0004] The purpose of this invention is to provide a quasi-e-ary neural network model, an electronic device, and a storage medium. The quasi-e-ary neural network model constructs neurons expressed by a combination of binary and ternary functions, and optimizes the full-precision activation function by linear combination. This reduces the storage space, computational complexity, and energy consumption of the neural network model, and effectively reduces information loss and improves the efficiency of the neural network compared with binary convolutional neural networks.
[0005] To achieve the above objectives, the present invention proposes the following technical solution:
[0006] The first aspect of this application provides a quasi-e-ary neural network model, which includes an input layer, a hidden layer, and an output layer.
[0007] The input layer is used to receive data;
[0008] The hidden layer comprises neurons, each of which has at least one sub-neuron. The sub-neuron can be represented as either a binary function g(x) or a ternary function t(x), and the activation function F(x) of the neuron is expressed as:
[0009] Where x is the input data, w is the weight, f(x) is the binary function g(x) or the ternary function t(x), and N is the number of sub-neurons in the neuron;
[0010] The output layer is used to output the model results.
[0011] In one possible implementation, the ratio of the number of binary functions g(x) and ternary functions t(x) in the activation function F(x) approximates the formula: e u+v ≈2 u +3 v ,
[0012] Where U is the number of binary functions g(x), V is the number of ternary functions t(x), and e is the natural constant.
[0013] In one possible implementation, the ratio of U to V is 1:3.
[0014] In one possible implementation, the binary function g(x) is expressed as: The ternary function t(x) is expressed as:
[0015] In one possible implementation, the activation function F(x) employs a pass-through gradient, the gradient of which is expressed as:
[0016] Where k is any value in [-1,1], and ∑D=[-1,1].
[0017] In one possible implementation, each of the sub-neurons is set to the three-valued function t(x).
[0018] In one possible implementation, a sub-neuron is provided in the neuron, and the weights associated with the neuron are binarized.
[0019] The second aspect of this application provides a method for constructing a neural network model, wherein the activation function F(x) described in any one of the first aspects of this application is used to replace at least a portion of the activation functions in the target model.
[0020] The third aspect of this application provides a data processing method that uses the quasi-e-ary neural network model described in any one of the first aspects of this application to process input data.
[0021] A fourth aspect of this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;
[0022] The memory is used to store computer programs;
[0023] When the processor executes the program stored in the memory, it uses any of the quasi-e-ary neural network models described in the first aspect of this application to obtain the output model result.
[0024] Beneficial effects:
[0025] As can be seen from the above technical solutions, this application constructs neurons expressed by a combination of binary and ternary functions. Binarization and ternary transformation of neurons simplify the calculation process, reduce computational complexity and model storage space, and use a near-e-ary ratio for binary and ternary functions. A linear combination is used to approximate the full-precision activation function, which solves the problem of excessive information loss and decreased accuracy when the activation function is binarized or ternary. This application reduces the computational complexity, energy consumption and storage space of the neural network, while ensuring the accuracy of the neural network and improving efficiency.
[0026] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below can be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other.
[0027] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description
[0028] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings, wherein:
[0029] Figure 1 A schematic diagram of a quasi-e-ary neural network model provided in an embodiment of the present invention;
[0030] Figure 2 This is a schematic diagram of a neuron in a quasi-e-ary neural network model provided in an embodiment of the present invention. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art.
[0032] The terms "first," "second," and similar words used in the specification and claims of this patent application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular forms of "an," "a," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. Terms such as "comprising" or "including" mean that the element or object preceding "comprising" encompasses the features, wholes, steps, operations, elements, and / or components listed following "comprising" or "including," and do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or collections thereof. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0033] The inventors of this application have observed that with the development of artificial intelligence and the Internet of Things (IoT), numerous embedded and edge devices in both industrial and consumer sectors are incorporating neural network models to enhance their intelligence. Examples include facial recognition and behavior monitoring in smart homes and defect detection in industrial production. However, due to their complex structure and massive parameter scale, neural network models consume enormous computational resources and have extremely high storage requirements. Deploying them in the cloud incurs high computational and storage costs. Furthermore, their practical application is limited by hardware constraints, particularly in resource-constrained scenarios such as mobile devices, embedded devices, and IoT devices, where their application becomes particularly difficult. The massive computational resource consumption also translates to high energy consumption, which in turn means shorter battery life, hindering the deployment of neural network models on mobile devices and resulting in higher operating costs in the cloud. Therefore, it is often necessary to quantize trained neural network models to reduce storage requirements and computational complexity. Binary quantization limits weights to +1 or -1, greatly simplifying the calculation process and reducing computational resource consumption, but this significantly slows down model progress. Besides quantization, there is also research on modifying neural network models to reduce hardware requirements. In these studies, binary neural networks (BNNs) compress weights and activation values by strictly limiting them to the binary range of -1 to 1 using binarization. Each weight and activation in the network is represented by a single bit, minimizing memory usage. Furthermore, binarization allows BNNs to replace cumbersome floating-point multiplication and addition operations with XNOR and Bitcount operations. Therefore, it significantly reduces memory consumption, improves execution speed, and lowers energy consumption. Subsequent training revealed that XNOR-Net, using 1-bit convolution operations, achieved a 58x speedup on CPUs and a 32x compression rate. However, due to information loss, model accuracy declined significantly, leading to accuracy degradation and loss of accuracy issues during training on complex image recognition tasks like ImageNet.
[0034] At least one embodiment of the disclosed model provides a quasi-e-ary neural network model, such as Figure 1 As shown, it includes an input layer, a hidden layer, and an output layer. The output layer is used to receive data. The hidden layer includes neurons, each with at least one sub-neuron. The sub-neuron can be represented as either a binary function g(x) or a ternary function t(x). The activation function F(x) of the neuron is expressed as:
[0035] Where x is the input data, w is the weight, f(x) is a binary function g(x) or a ternary function t(x), N is the number of sub-neurons in the neuron; the output layer is used to output the model results.
[0036] In this embodiment, the activation function used for the neuron is a linear combination of the binary function g(x) and the ternary function t(x). This design offers significant advantages in terms of activation function storage and computation, and can largely maintain the high compression ratio and high computational efficiency of binary neural networks. Specifically, by using a linear weighted fitting method, the full-precision model can be approximated more effectively, thereby ensuring that the accuracy meets the requirements of practical applications.
[0037] Meanwhile, the introduction of a ternary function t(x) allows the neuron to approximate base e in computation, primarily by balancing energy consumption and accuracy. In practical applications, energy consumption is a critical consideration for electronic devices, while accuracy is equally indispensable. This approach effectively reduces energy consumption while maintaining a certain level of accuracy, supporting the efficient operation of electronic devices.
[0038] In some examples, to balance the computational efficiency and energy consumption of the neuron's activation function, the ratio of the binary function g(x) or the ternary function t(x) is directly approximated by the formula: e u+v ≈2 u +3 v In this equation, U represents the number of binary functions g(x), V represents the number of ternary functions t(x), and e is the natural constant. When the ratio of U to V gradually approaches the specific numerical relationship presented by the above equation, the activation function will move closer to the e-ary system. In this state, the activation function can achieve an excellent balance between energy consumption and accuracy. On the one hand, regarding energy consumption, it can effectively control and optimize the energy consumption of electronic devices during operation by using the e-ary activation function characteristic, thus extending the battery life of electronic devices. On the other hand, regarding accuracy, this e-ary activation function does not reduce energy consumption at the expense of accuracy. It can balance the relationship between energy consumption and accuracy while maintaining a certain level of accuracy, ensuring that it meets the needs of various complex applications in different application scenarios. By constructing different ratios of binary and ternary functions to reduce computational requirements and energy consumption while meeting accuracy requirements in the neural network model, it was found that when the ratio is close to the formula described above, the constructed neural network model can effectively reduce energy consumption while ensuring accuracy.
[0039] The enumeration method reveals that the closest approximation to base e is achieved when the ratio of U to V is 1:3. A preferred embodiment of this disclosure is, for example... Figure 2 As shown, the neuron includes 4 sub-neurons, one of which is represented by a binary function g(x), and the other three are represented by a ternary function t(x).
[0040] There are many ways to express binary and ternary functions. In some examples disclosed herein, the binary function g(x) is represented as follows:
[0041]
[0042] The ternary function t(x) is expressed as:
[0043]
[0044] Because of the oversaturation of activation values in binary neural networks, gradient vanishing occurs, resulting in insufficient gradient estimation for parameter learning and making them prone to getting trapped in local suboptimal solutions. The embodiments of this application employ a pass-through gradient activation function, expressed as: Where k takes any value from -1 to 1, ∑D=[-1,1], and when the cumulative value of D exceeds the upper and lower limits, the corresponding upper and lower limits are taken.
[0045] In some specific examples, each sub-neuron within a neuron can be represented by a three-valued function t(x). When this is done, the model's accuracy can be further improved. This is because the three-valued function t(x) introduces more possibilities and flexibility into the model, allowing neurons to more accurately reflect the actual situation when processing information and performing calculations. In various complex application scenarios, improved accuracy means the model can better adapt to different task requirements, providing users with more reliable and accurate results.
[0046] In other embodiments, the neuron has only one child neuron. In this case, the weights associated with the neuron are binarized. In this example, the low bit-width of both the weights and activation function has several positive effects. First, it excels in reducing energy consumption. Low bit-width weights and activation functions mean less energy is required during computation, which is crucial for electronic devices, especially mobile devices. It can extend device battery life and reduce energy consumption. Second, it achieves extremely high computational efficiency. The low bit-width characteristic makes the computation process simpler and more efficient, reducing unnecessary computational complexity and time overhead. Whether in large-scale data processing or real-time applications, this high computational efficiency can provide users with a faster and smoother experience. In summary, this approach achieves extremely high computational efficiency while reducing energy consumption.
[0047] It should be noted that, in practical applications, the quasi-e-ary neural network model provided in the embodiments of this disclosure can quantize the weight parameters of the neural network model according to actual needs.
[0048] At least one embodiment of this disclosure also provides a method for constructing a neural network model, in which the activation function F(x) provided in any embodiment of this disclosure is used to replace at least a portion of the activation functions in the target model. In practical applications, while preserving the accuracy of the target model, the storage space of the target model is compressed, and the energy consumption of the target model is reduced. This measure has significant practical implications; compressing the storage space of the target model can reduce the demand for storage devices, lower storage costs, and also facilitate model transmission and deployment. Furthermore, reducing the energy consumption of the target model is also one of our important goals. By optimizing the model structure and using low-energy computing methods and hardware devices, the energy consumption of the target model during operation can be significantly reduced. This not only helps reduce energy waste and lower operating costs, but is particularly crucial for energy-sensitive applications such as mobile devices and embedded systems.
[0049] In summary, to deploy a neural network model on the terminal, it is necessary to compress the target model's storage space and reduce its power consumption while preserving the target model's accuracy. Currently, many AI consumer electronics offer functions such as object recognition and character recognition, which are mostly achieved by sending information collected by the terminal to a cloud model for processing, and then the terminal receives and displays the results. A significant reason for this is the hardware limitations of embedded devices, making it difficult to deploy neural network models with sufficient precision and low power consumption on the edge. The neural network model constructed in this invention balances the requirements of accuracy and power consumption, meeting the requirements of embedded devices for small storage space and low power consumption of neural network models, while its efficiency approaches that of a full-precision model.
[0050] At least one embodiment of this disclosure also provides a data processing method, which includes: processing input data using a quasi-e-ary neural network model provided in any embodiment of this disclosure to obtain an output result.
[0051] For example, in some cases, the type of input data depends on the object the model is processing. Depending on the object the model is processing, the input data can include images, text, speech, etc. For instance, neural network models such as ResNet, MobileNet-V1, MobileNet-V2, and VGG-Net, and their target quantization models, can use images as input data.
[0052] For example, output data can represent the model's inference and prediction results based on input data. Taking neural network models such as ResNet, MobileNet-V1, and VGG-Net, and their target quantization models, as examples, their output data can represent the classification results of images (i.e., input data).
[0053] For example, in some cases, the quasi-e-ary neural network model can be deployed in mobile and embedded devices such as smartphones, tablets, and in-vehicle navigation systems, so that these mobile and embedded devices can perform the data processing methods described above.
[0054] The following section uses the ResNet-18 model as an example to construct a new model, and the table below illustrates the efficiency comparison between this disclosure and the full-precision model on the COCO dataset. ResNet-18 is an 18-layer model in the ResNet (Residual Network) series, proposed by Microsoft Research Asia in 2015. This model is widely used in computer vision tasks such as image classification and object detection.
[0055] Model Accuracy (%) Model size (MB) Time elapsed (ms) Energy consumption (W) ResNet18 93.6 46.7 1418.94 16.3 2B-Net 87.5 3.7 145.8 0.9 3B-Net 91.9 5.6 192.3 1.2 E-Net 90.6 4.3 176.1 1.1
[0056] 2B-Net is a binary combined neural network. The model construction process replaces the activation function in the ResNet-18 model with a combination of binary functions g(x). The four binary function g(x) sub-neurons are generated by combining them in the manner described in the above embodiment. 3B-Net is a ternary function t(x) combined neural network. The model construction process replaces the activation function in the ResNet-18 model with a ternary combination. The four ternary neurons are generated by combining them in the manner described in the above embodiment. E-Net is a combined neural network of binary functions g(x) and ternary functions t(x). The model construction process replaces the activation function in the ResNet-18 model with one binary function g(x) and three ternary functions t(x), and they are generated by combining them in the manner described in the above embodiment.
[0057] As shown in the table above, the accuracy of the constructed model provided by the embodiments of this disclosure is close to that of the original model, and the computational efficiency and energy consumption of the model are better than those of the original model. At the same time, the model is greatly compressed.
[0058] The technical effects of the data processing method provided in the embodiments of this disclosure can be found in the corresponding descriptions of the neural network model construction method in the above embodiments, and will not be repeated here.
[0059] This application also provides a computer program product or computer program that may include computer instructions, which may be stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the description of the quasi-e-ary neural network model in the preceding embodiments; therefore, this will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here. For technical details not disclosed in the embodiments of the computer program product or computer program involved in this application, please refer to the description of the method embodiments of this application.
[0060] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, hardware + program embodiments are relatively simple in description because they are fundamentally similar to method embodiments; relevant parts can be referred to the descriptions in the method embodiments.
[0061] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0062] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded.
[0063] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing the embodiments of this specification, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.
[0064] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.
[0065] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0066] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0067] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0068] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0069] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0070] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0071] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0072] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0073] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0074] The above description is merely an embodiment of the present specification and is not intended to limit the embodiments of the present specification. For those skilled in the art, various modifications and variations can be made to the embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of the present specification should be included within the scope of the claims of the embodiments of the present specification.
[0075] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.
Claims
1. A quasi-e-ary neural network model, characterized in that: The quasi-e-ary neural network model includes an input layer, a hidden layer, and an output layer. The input layer is used to receive data; The hidden layer comprises neurons, each of which has at least one sub-neuron. The sub-neuron can be represented as either a binary function g(x) or a ternary function t(x), and the activation function F(x) of the neuron is expressed as: Where x is the input data, w is the parameter, f(x) is the binary function g(x) or the ternary function t(x), and N is the number of sub-neurons in the neuron; The output layer is used to output the model results.
2. The quasi-e-ary neural network model according to claim 1, characterized in that: The ratio of the number of binary functions g(x) and ternary functions t(x) in the activation function F(x) approximates the formula: e u+v ≈2 u +3 v , Where U is the number of binary functions g(x), V is the number of ternary functions t(x), and e is the natural constant.
3. The quasi-e-ary neural network model according to claim 2, characterized in that: The ratio of U to V is 1:
3.
4. The quasi-e-ary neural network model according to claim 1, characterized in that: The binary function g(x) is expressed as: The ternary function t(x) is expressed as:
5. The quasi-e-ary neural network model according to claim 1, characterized in that: The activation function F(x) uses a pass-through gradient, and the gradient of the activation function F(x) is expressed as: Where k takes any value in [-1,1], and ∑D=[-1,1].
6. The quasi-e-ary neural network model according to claim 1, characterized in that: Each sub-neuron is set to the three-valued function t(x).
7. The quasi-e-ary neural network model according to claim 6, characterized in that: A sub-neuron is set in the neuron, and the weights associated with the neuron are binarized.
8. A method for constructing a neural network model, characterized in that... This includes replacing at least a portion of the activation functions in the target model with the activation function F(x) as described in any one of claims 1-7.
9. A data processing method, characterized in that... This includes processing the input data using the quasi-e-ary neural network model described in any one of claims 1-7.
10. An electronic device, characterized in that... It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it uses the quasi-e-ary neural network model described in any one of claims 1-7 to obtain the output model result.