Activation module, neural network and data processing method and device
By constructing lightweight activation modules and neural networks, the problem of high computational load in image processing models on terminal devices is solved, enabling rapid training and efficient processing of low-level visual tasks.
Patent Information
- Application Number
- CN202410705412.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-31
- Publication Date
- 2025-12-02
AI Technical Summary
Existing AI models for image processing require a large amount of computation and are difficult to run effectively on terminal devices such as smartphones, especially for low-level visual tasks where the computational power requirements are difficult to meet.
Design an activation module and neural network to construct a lightweight neural network through channel expansion and rearrangement, comprising sequentially connected activation blocks and separable convolutional layers for image and audio feature processing.
It achieves lightweight neural networks, improves model representation capabilities, reduces computational load, and is suitable for rapid training and deployment on terminal devices to handle low-level vision tasks such as super-resolution and denoising.
Smart Images
Figure CN121052298A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, specifically to an activation module, a neural network, a data processing method, and an apparatus. Background Technology
[0002] With the rapid development of artificial intelligence (AI) technology, many fields of work have integrated AI, significantly improving both effectiveness and efficiency. For example, the integration of AI into image processing has enabled AI to participate in or complete many image processing tasks, resulting in significantly improved effectiveness and efficiency.
[0003] However, the artificial intelligence models used for image processing in related technologies are quite complex, computationally intensive, and place very high demands on equipment. In particular, low-level vision tasks mostly run on terminals such as smartphones, and the computing power of these terminals is often insufficient to meet the computational requirements of the aforementioned artificial intelligence models. Summary of the Invention
[0004] To overcome the problems existing in the related technologies, the present disclosure provides an activation module, a neural network, a data processing method and apparatus to solve the defects in the related technologies.
[0005] According to a first aspect of the present disclosure, an activation module for use in a neural network is provided. The activation module includes at least one activation block connected in sequence. The activation block is used to perform channel expansion on input features, divide all the obtained channels into multiple channel groups, and determine one channel of the output feature according to each channel group.
[0006] For each channel of the output feature, the element value at each position in the channel is determined based on multiple element values at the same position within the corresponding channel group.
[0007] The number of channel groups is the same as the number of channels in the input feature;
[0008] The input features include image features and audio features.
[0009] In one embodiment of this disclosure, for each channel of the output feature, the element value at each position on the channel is the maximum value among multiple element values at the same position within the corresponding channel group; or,
[0010] For each channel of the output feature, the element value at each position in the channel is the minimum value among multiple element values at the same position within the corresponding channel group; or,
[0011] For each channel of the output feature, the element value at each position in the channel is the average of multiple element values at the same position within the corresponding channel group.
[0012] In one embodiment of this disclosure, when the activation block is used to perform channel expansion on the input feature, it is used to: expand the channels of the input feature by N times;
[0013] When the activation block is used to divide all the obtained channels into multiple channel groups, it is used to: divide all the channels obtained by channel expansion into N channel sets, rearrange the channels of the N channel sets to obtain multiple channel groups with the same number of channels as the input feature, and each channel group includes N channels.
[0014] In one embodiment of this disclosure, the activation block is used to rearrange the channels of the N channel sets to obtain multiple channel groups with the same number of channels as the input feature, for use in...
[0015] Channels in the same order within the N channel sets are grouped into a channel group.
[0016] In one embodiment of this disclosure, when the activation block is used to perform channel expansion on the input features, it is used for:
[0017] The input features are subjected to a convolution operation to expand the channels of the input features.
[0018] In one embodiment of this disclosure, the activation module further includes a separable convolutional layer for performing separable convolution operations on the input features and inputting the resulting output features into the first activation block.
[0019] According to a second aspect of the present disclosure, a neural network is provided, the neural network comprising:
[0020] Convolutional layers are used to perform convolution operations on the data to be processed, which includes image data and audio data;
[0021] At least one activation module as described in any embodiment of the first aspect is connected in sequence, wherein the first activation module takes the output of the convolutional layer as input.
[0022] In one embodiment of this disclosure, the neural network is used to process low-level vision tasks.
[0023] In one embodiment of this disclosure, the underlying vision task includes a super-resolution task;
[0024] The neural network also includes a pixel rearrangement module, which takes the output of the last activation module as input.
[0025] According to a third aspect of the present disclosure, a data processing method is provided, the method comprising:
[0026] The neural network described in any embodiment of the second aspect is used to process the data to be processed, wherein the data to be processed includes image data and audio data.
[0027] According to a fourth aspect of the present disclosure, a data processing apparatus is provided, the apparatus including a processing module for:
[0028] The neural network described in any embodiment of the second aspect is used to process the data to be processed, wherein the data to be processed includes image data and audio data.
[0029] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the third aspect.
[0030] According to a sixth aspect of the present disclosure, an electronic device is provided, the electronic device including a memory and a processor, the memory being configured to store computer instructions executable on the processor, and the processor being configured to implement the method described in the third aspect when executing the computer instructions.
[0031] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the third aspect.
[0032] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:
[0033] The activation block in the activation module provided in this embodiment can expand the input features into channels, divide all the resulting channels into multiple channel groups, and determine one channel of the output feature according to each channel group. For each channel of the output feature, the element value at each position in the channel is determined based on multiple element values at the same position within the corresponding channel group. The number of channel groups is the same as the number of channels in the input features, which include image features and audio features. Therefore, the activation module can increase the model representation capability of the neural network. Furthermore, the activation module has a simple structure, low computational cost, and can be used to build lightweight neural networks with fast training convergence speed. The constructed neural network is particularly suitable for deployment on terminals to handle low-level visual tasks such as super-resolution and denoising. Attached Figure Description
[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0035] Figure 1This is a schematic diagram of the structure of the activation module shown in an exemplary embodiment of this disclosure;
[0036] Figure 2 This is a schematic diagram of the structure of a neural network shown in an exemplary embodiment of this disclosure;
[0037] Figure 3 This is a structural block diagram of an electronic device illustrated in an exemplary embodiment of the present disclosure. Detailed Implementation
[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0039] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.
[0040] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0041] With the rapid development of artificial intelligence (AI) technology, many fields of work have integrated AI, significantly improving both effectiveness and efficiency. For example, the integration of AI into image processing has enabled AI to participate in or complete many image processing tasks, resulting in significantly improved effectiveness and efficiency.
[0042] However, the artificial intelligence models used for image processing in related technologies are quite complex, computationally intensive, and place very high demands on equipment. In particular, low-level vision tasks mostly run on terminals such as smartphones, and the computing power of these terminals is often insufficient to meet the computational requirements of the aforementioned artificial intelligence models.
[0043] Low-level vision tasks are image processing tasks that deal with pixel-level images. They transform images from raw signals or a certain observation state into what people want to see, usually clear, natural images. Examples include denoising, super-resolution, and image enhancement.
[0044] Based on this, in a first aspect, at least one embodiment of this disclosure provides an activation module applied to a neural network, please refer to the appendix. Figure 1 The activation module includes at least one activation block connected in sequence. The activation block is used to perform channel expansion on the input features, divide all the obtained channels into multiple channel groups, and determine one channel of the output feature according to each channel group.
[0045] The input features include image features and audio features.
[0046] The number of channel groups is the same as the number of channels in the input feature.
[0047] Specifically, for each channel of the output feature, the element value at each position in the channel is determined based on multiple element values at the same position within the corresponding channel group. For example, for each channel of the output feature, the element value at each position in the channel is the maximum value among multiple element values at the same position within the corresponding channel group; or, for each channel of the output feature, the element value at each position in the channel is the minimum value among multiple element values at the same position within the corresponding channel group; or, for each channel of the output feature, the element value at each position in the channel is the average value among multiple element values at the same position within the corresponding channel group.
[0048] Preferably, for each channel of the output feature, the element value at each position in the channel is the maximum value among multiple element values at the same position within the corresponding channel group. In this case, the activation block can be called an Expand Max-out Activation block; Appendix Figure 1 The example shown is an activation block using Expand Max-out Activation, but this is not a limitation on the type of activation block. In this case, the activation module can be called a Series max-out block (SMOBlk).
[0049] For example, when the activation block is used to perform channel expansion on the input features, it can perform a convolution operation on the input features to expand the channels. For instance, the activation block may contain a convolutional layer for performing convolution operations on the input features.
[0050] As another example, when the activation block is used to expand the channels of the input feature, it can expand the channels of the input feature by N times.
[0051] Based on the above expansion method, when the activation block is used to divide all the obtained channels into multiple channel groups, it can divide all the channels obtained from channel expansion into N channel sets, and rearrange the channels in the N channel sets to obtain multiple channel groups with the same number of channels as the input feature, each channel group including N channels. For example, channels in the same order within the N channel sets can be grouped into one channel group.
[0052] The above grouping method ensures that the number of channel groups is the same as the number of channels in the input features. This allows for the generation of one channel for the output features based on each channel group, thus ensuring that the number of channels in the input features is the same as the number of channels in the output features.
[0053] For example, appendix Figure 1 In the activation block shown, the convolutional layer expands the 64-channel input features into 64*3 channels; then, it divides the 64*3 channels into 3 channel sets; then, it performs a channel shuffle on the 3 channel sets, that is, it groups the 3 channels together with channels of the same order, thus obtaining 64 channel groups, each containing 3 channels; then, for each channel group, it maxes out the maximum value of the 3 channel values at each position, thus obtaining the element value at each position of the corresponding output channel; finally, it concatenates the 64 output channels together to obtain the 64-channel output features.
[0054] The activation block in the activation module provided in this embodiment can expand the input features into channels, divide all the resulting channels into multiple channel groups, and determine one channel of the output feature according to each channel group. For each channel of the output feature, the element value at each position in the channel is determined based on multiple element values at the same position within the corresponding channel group. The number of channel groups is the same as the number of channels in the input features, which include image features and audio features. Therefore, the activation module can increase the model representation capability of the neural network. Furthermore, the activation module has a simple structure, low computational cost, and can be used to build lightweight neural networks with fast training convergence speed. The constructed neural network is particularly suitable for deployment on terminals to handle low-level visual tasks such as super-resolution and denoising.
[0055] In some embodiments of this disclosure, please refer to the appendix. Figure 1The activation module further includes a separable convolutional layer Dconv, which is used to perform separable convolution operations on the input features and input the resulting output features into the first activation block.
[0056] In this embodiment, the separable convolutional layer extracts features on each channel, thereby giving the activation module a better feature extraction capability. Moreover, the computational cost of the separable convolutional layer is small, making the activation module suitable for building lightweight neural networks.
[0057] Secondly, at least one embodiment of this disclosure provides a neural network, please refer to the appendix. Figure 2 The neural network includes: a convolutional layer Conv for performing convolution operations on data to be processed, wherein the data to be processed includes image data and audio data; and at least one activation module connected in sequence as described in any embodiment of the first aspect, wherein the first activation module takes the output of the convolutional layer as input. (Appendix) Figure 2 The neural network in the example includes 5 activation modules; and the activation modules shown are SMOBlk, but this is not a limitation on the types of activation modules.
[0058] For example, the neural network is used to process low-level vision tasks, such as denoising and super-resolution.
[0059] It should be understood that, please refer to the appendix. Figure 2 When the neural network is used to process super-resolution tasks (i.e., super-resolution tasks), the neural network may further include a pixel shuffle module, which takes the output of the last activation module as input.
[0060] The neural network provided in this embodiment is a lightweight model with less computation and faster convergence speed. It can perform well in low-level visual tasks such as image and video denoising, enhancement, restoration, compression, HDR, and super-resolution. For example, it has good image processing effect and high efficiency, and is suitable for deployment on terminals with limited computing power such as smartphones.
[0061] Thirdly, at least one embodiment of this disclosure provides a data processing method, the method comprising: processing data to be processed using a neural network as described in any embodiment of the second aspect, wherein the data to be processed includes image data and audio data.
[0062] According to a fourth aspect of the present disclosure, a data processing apparatus is provided, the apparatus including a processing module for:
[0063] The neural network described in any embodiment of the second aspect is used to process the data to be processed, wherein the data to be processed includes image data and audio data.
[0064] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the third aspect.
[0065] According to the sixth aspect of the embodiments of this disclosure, please refer to the appendix. Figure 3 The diagram illustrates, for example, a block diagram of an electronic device. For instance, device 300 could be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.
[0066] Reference Figure 3 The device 300 may include one or more of the following components: a processing component 302, a memory 304, a power supply component 306, a multimedia component 308, an audio component 310, an input / output (I / O) interface 312, a sensor component 314, and a communication component 316.
[0067] Processing component 302 typically controls the overall operation of device 300, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 302 may include one or more modules to facilitate interaction between processing component 302 and other components. For example, processing component 302 may include a multimedia module to facilitate interaction between multimedia component 308 and processing component 302.
[0068] Memory 304 is configured to store various types of data to support the operation of device 300. Examples of this data include instructions for any application or method operating on device 300, contact data, phonebook data, messages, pictures, videos, etc. Memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0069] The power supply component 306 provides power to the various components of the device 300. The power supply component 306 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 300.
[0070] Multimedia component 308 includes a screen that provides an output interface between the device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 308 includes a front-facing camera and / or a rear-facing camera. When the device 300 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
[0071] Audio component 310 is configured to output and / or input audio signals. For example, audio component 310 includes a microphone (MIC) configured to receive external audio signals when device 300 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 304 or transmitted via communication component 316. In some embodiments, audio component 310 also includes a speaker for outputting audio signals.
[0072] I / O interface 312 provides an interface between processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.
[0073] Sensor assembly 314 includes one or more sensors for providing status assessments of various aspects of device 300. For example, sensor assembly 314 may detect the on / off state of device 300, the relative positioning of components such as the display and keypad of device 300, changes in the position of device 300 or a component of device 300, the presence or absence of user contact with device 300, the orientation or acceleration / deceleration of device 300, and temperature changes of device 300. Sensor assembly 314 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 314 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 314 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.
[0074] Communication component 316 is configured to facilitate wired or wireless communication between device 300 and other devices. Device 300 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 316 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0075] In an exemplary embodiment, the apparatus 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the data processing method described above.
[0076] In a seventh aspect, this disclosure also provides a non-transitory computer-readable storage medium including instructions, such as a memory 304 including instructions, which can be executed by a processor 320 of the device 300 to complete the above-described data processing method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.
[0077] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.
[0078] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. An activation module applied to a neural network, characterized in that, The activation module includes at least one activation block connected in sequence. The activation block is used to perform channel expansion on the input features, divide all the obtained channels into multiple channel groups, and determine one channel of the output feature according to each channel group. For each channel of the output feature, the element value at each position in the channel is determined based on multiple element values at the same position within the corresponding channel group. The number of channel groups is the same as the number of channels in the input feature; The input features include image features and audio features.
2. The activation module applied to a neural network according to claim 1, characterized in that, For each channel of the output feature, the element value at each position in the channel is the maximum value among multiple element values at the same position within the corresponding channel group; or, For each channel of the output feature, the element value at each position in the channel is the minimum value among multiple element values at the same position within the corresponding channel group; or, For each channel of the output feature, the element value at each position in the channel is the average of multiple element values at the same position within the corresponding channel group.
3. The activation module applied to a neural network according to claim 1, characterized in that, When the activation block is used to perform channel expansion on the input feature, it is used to: expand the channels of the input feature by N times; When the activation block is used to divide all the obtained channels into multiple channel groups, it is used to: divide all the channels obtained by channel expansion into N channel sets, rearrange the channels of the N channel sets to obtain multiple channel groups with the same number of channels as the input feature, and each channel group includes N channels.
4. The activation module applied to a neural network according to claim 3, characterized in that, The activation block is used to rearrange the channels of the N channel sets to obtain multiple channel groups with the same number of channels as the input feature, and is used for Channels in the same order within the N channel sets are grouped into a channel group.
5. The activation module applied to a neural network according to claim 1, characterized in that, The activation block is used to perform channel expansion on the input features for: The input features are subjected to a convolution operation to expand the channels of the input features.
6. The activation module applied to a neural network according to any one of claims 1 to 5, characterized in that, The activation module also includes a separable convolutional layer, which is used to perform separable convolution operations on the input features and input the resulting output features into the first activation block.
7. A neural network, characterized in that, The neural network includes: Convolutional layers are used to perform convolution operations on the data to be processed, which includes image data and audio data; At least one activation module as described in any one of claims 1 to 6 is connected in sequence, wherein the first activation module takes the output of the convolutional layer as input.
8. The neural network according to claim 7, characterized in that, The neural network is used to process low-level vision tasks.
9. The neural network according to claim 8, characterized in that, The underlying vision task includes super-resolution tasks; The neural network also includes a pixel rearrangement module, which takes the output of the last activation module as input.
10. A data processing method, characterized in that, The method includes: The neural network according to any one of claims 7 to 9 is used to process the data to be processed, wherein the data to be processed includes image data and audio data.
11. A data processing apparatus, characterized in that, The device includes a processing module for: The neural network according to any one of claims 7 to 9 is used to process the data to be processed, wherein the data to be processed includes image data and audio data.
12. A computer program product comprising a computer program / instructions, characterized in that, When the computational program / instructions are executed by the processor, they implement the steps of the method of claim 10.
13. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store computer instructions that can run on the processor, and the processor being used to implement the method of claim 10 when executing the computer instructions.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method of claim 10.