Data processing method and related device

Abstract

The invention discloses a data processing method, and the method comprises the steps: obtaining a mask of input data through a prediction network according to the input data of a convolutional layer, and the mask comprises a mask of a channel dimension and a mask of a spatial dimension; obtaining a prefix sum on the channel dimension according to the mask of the channel dimension; according to the mask of the spatial dimension, obtaining a prefix sum on the spatial dimension when the input data is subjected to one-time sliding window convolution through the convolution kernel of the convolution layer; and acting a mask on the input data according to a fusion result of the prefix on the channel dimension and the prefix sum on the spatial dimension, and carrying out sliding window convolution processing on the masked input data through a convolution kernel of a convolution layer. According to the method, the prefix sum of the channel dimension is calculated in advance, so that only the prefix sum of the spatial dimension needs to be calculated during window sliding every time, and the mask acts on the input data by utilizing the fusion result of the prefix sum of the channel dimension and the prefix sum of the spatial dimension, so that the demand of computing resources and the computing delay are greatly reduced.

Description

Technical Field

[0001] This application relates to the technical field of artificial intelligence (AI), and particularly to a data processing method, a computer-readable storage medium, and a computer program product. Background Art

[0002] In order to improve the fault tolerance of a machine learning model, fault tolerance training can be added to the training of the dynamic inference module. The so-called fault tolerance can be understood as: when the hardware processes data through the model, when part of the output result of the intermediate layer of the network is missing due to hardware reasons (that is, read / write noise is generated during hardware operation), the model can still have a high processing accuracy. During training, simulated random noise is injected into the model to approximate the read / write noise generated during actual hardware operation, so that the mask generator for dynamic inference can adapt to the corresponding intensity of read / write noise to a certain extent and give the optimal mask. During the inference stage, the spatial mask generator and the channel mask generator respectively give the corresponding masks, which act on the output of the network layer (such as the feature map) and the weights of the model.

[0003] Among them, when the mask acts on the output of the intermediate layer (that is, the input of the next network layer), it is necessary to calculate the prefix sum through the hardware based on the mask. Through the prefix sum, the mask can act on the data, and then the network layer can process the masked data.

[0004] The data can include data in the channel dimension and data in the spatial dimension. The mask can include masks in the channel dimension and the spatial dimension. When the convolutional layer performs a convolutional operation, it can perform a sliding window convolutional operation on the input data through the convolutional kernel. In the prior art, the prefix sum (including the prefix sum in the channel dimension and the prefix sum in the spatial dimension) needs to be calculated through the mask every time the window slides. However, calculating the prefix sum requires hardware units to implement. The prefix sum including the prefix sum in the channel dimension and the prefix sum in the spatial dimension is three-dimensional data (that is, a large amount of data). Therefore, it requires a large amount of computing resources and will cause a large computing delay. Summary of the Invention

[0005] In a first aspect, the present application provides a data processing method, the method comprising: obtaining a mask of the input data through a prediction network according to the input data of the convolutional layer; the input data comprising data in a channel dimension and a spatial dimension; the mask comprising a mask in the channel dimension and a mask in the spatial dimension; obtaining a prefix sum in the channel dimension according to the mask in the channel dimension; obtaining a prefix sum in the spatial dimension when performing a single sliding window convolution on the input data by means of the convolutional kernel of the convolutional layer according to the mask in the spatial dimension; and acting the mask on the input data according to a fusion result of the prefix sum in the channel dimension and the prefix sum in the spatial dimension, and performing processing of the sliding window convolution on the masked input data by means of the convolutional kernel of the convolutional layer.

[0006] For the same input data (such as the same picture), the channel mask is constant, and only the spatial mask changes with the sliding window. Therefore, the channel mask can be preprocessed to calculate the prefix sum in the channel dimension in advance. Thus, each time the sliding window is performed, only the prefix sum in the spatial dimension needs to be calculated, and the mask is acted on the input data by using the fusion result of the prefix sum in the channel dimension and the prefix sum in the spatial dimension. Equivalently, when the convolutional layer performs each sliding window on the same input data, only the pre-calculated prefix sum in the channel dimension needs to be reused, and only the prefix sum in the spatial dimension needs to be recalculated, greatly reducing the requirements for computing resources and the computing latency.

[0007] In a possible implementation, the fusion result is the Cartesian product of the prefix sum in the channel dimension and the prefix sum in the spatial dimension.

[0008] In a possible implementation, the input data is a multi-channel feature map.

[0009] In a possible implementation, the method further comprises: obtaining a prefix sum in the spatial dimension when performing another sliding window convolution on the input data by means of the convolutional kernel of the convolutional layer according to the mask in the spatial dimension; and acting the mask on the input data according to a fusion result of the prefix sum in the channel dimension and the prefix sum in the spatial dimension during the another sliding window convolution, and performing processing of the another sliding window convolution on the masked input data by means of the convolutional kernel of the convolutional layer.

[0010] That is to say, for a convolutional layer, during different sliding window convolutions, the pre-calculated prefix sum in the channel dimension can be reused without having to recalculate, thereby reducing the requirements for computing resources and the computing latency.

[0011] In a possible implementation, obtaining the prefix sum in the channel dimension according to the mask in the channel dimension includes: obtaining the prefix sum in the channel dimension through a first hardware unit according to the mask in the channel dimension; obtaining the prefix sum in the spatial dimension when performing a single sliding window convolution on the input data by the convolution kernel of the convolution layer according to the mask in the spatial dimension includes: obtaining the prefix sum in the channel dimension through a second hardware unit according to the mask in the spatial dimension, where the first hardware unit and the second hardware unit are different.

[0012] In a possible implementation, the method further includes: fusing the prefix sum in the channel dimension and the prefix sum in the spatial dimension through a third hardware unit to obtain the fusion result, where the third hardware unit is different from the first hardware unit and the second hardware unit.

[0013] In a possible implementation, the method further includes: processing the image data through a first network to obtain the input data of the convolution layer; processing the output data of the convolution layer through a second network to obtain the processing result of the image data, where the first network, the second network, and the convolution layer belong to a target network, and the target network is used to process the image data for a target task to obtain the processing result, and the target task is one of image classification, target recognition, text recognition in an image, image segmentation, and image enhancement.

[0014] In a possible implementation, the method is applied to the data processing process of a target network; the target network is obtained through neural network search; the method further includes: obtaining a plurality of candidate neural networks; selecting the target network from the plurality of candidate neural networks according to the performance of each candidate neural network; the performance is represented by the mean of the performance distribution and the standard deviation of the performance distribution.

[0015] That is to say, in a possible implementation, the method is applied to the data processing process of a target network; the target network is obtained through neural network search; obtaining a plurality of candidate neural networks; selecting the target network from the plurality of candidate neural networks according to the performance of each candidate neural network; the performance is represented by the mean of the performance distribution and the standard deviation of the performance distribution.

[0016] In the embodiments of the present application, through the design of a performance predictor for fault-tolerant architecture search, the mean of the model performance can be better predicted, and the standard deviation can also be effectively predicted, so that the obtained model has both a good average level and can perform stably.

[0017] In a possible implementation, the mean value and the standard deviation are determined by Bayesian inference and Monte Carlo sampling.

[0018] In the embodiments of the present application, the Monte Carlo random inactivation method can be applied to the predictor. Each forward inference is equivalent to a single sampling of the fault tolerance performance of the input architecture. For this input architecture, when the number of inferences is large enough, the sampled performance values output show the target fault tolerance performance distribution. An additional prediction head can be used to map the architecture hidden representation to the "standard deviation" of the fault tolerance performance evaluation of this architecture. The larger this value is, the higher the degree of diffusion. In particular, this standard deviation does not directly correspond to the true performance distribution standard deviation, but is only a measure that can be learned by the model.

[0019] In a second aspect, the present application provides a data processing device, and the device includes:

[0020] A processing module, configured to obtain a mask of the input data through a prediction network according to the input data of the convolutional layer; the input data includes data in the channel dimension and the spatial dimension; the mask includes a mask in the channel dimension and a mask in the spatial dimension;

[0021] A first hardware unit, configured to obtain a prefix sum in the channel dimension according to the mask in the channel dimension;

[0022] A second hardware unit, configured to obtain a prefix sum in the spatial dimension when performing a single sliding window convolution on the input data by the convolution kernel of the convolutional layer according to the mask in the spatial dimension;

[0023] The processing module is further configured to, according to a fusion result of the prefix sum in the channel dimension and the prefix sum in the spatial dimension, apply the mask to the input data, and perform the sliding window convolution processing on the masked input data through the convolution kernel of the convolutional layer.

[0024] In a possible implementation, the fusion result is the Cartesian product of the prefix sum in the channel dimension and the prefix sum in the spatial dimension.

[0025] In a possible implementation, the input data is a multi-channel feature map.

[0026] In a possible implementation, the processing module is further configured to:

[0027] Process the image data through a first network to obtain the input data of the convolutional layer;

[0028] Process the output data of the convolutional layer through a second network to obtain a processing result of the image data. The first network, the second network, and the convolutional layer belong to a target network, and the target network is used to process the image data for a target task to obtain the processing result. The target task is one of image classification, target recognition, text recognition in an image, image segmentation, and image enhancement.

[0029] In a possible implementation, the apparatus is applied to the data processing process of the target network;

[0030] The target network is obtained by neural network search;

[0031] The processing module is further configured to:

[0032] Obtain a plurality of candidate neural networks;

[0033] Select the target network from the plurality of candidate neural networks according to the performance of each candidate neural network; the performance is represented by the mean of the performance distribution and the standard deviation of the performance distribution.

[0034] In a possible implementation, the mean and the standard deviation are determined by Bayesian inference and Monte Carlo sampling.

[0035] In a third aspect, the present application provides a chip, including at least one processing unit and an interface circuit. The interface circuit is configured to provide program instructions or data for the at least one processing unit, and the at least one processing unit is configured to execute the program instructions to implement the method according to any one of the first aspect. The at least one processing unit includes a first hardware unit and a second hardware unit. The first hardware unit is configured to calculate the prefix sum in the channel dimension, and the second hardware unit is configured to calculate the prefix sum in the spatial dimension.

[0036] In a fourth aspect, an embodiment of the present application provides a data processing apparatus, which may include a memory, a processor, and a bus system. The memory is configured to store a program, and the processor is configured to execute the program in the memory to execute the method according to the first aspect and any of its optional aspects as described above.

[0037] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on a computer, it causes the computer to execute the method according to the first aspect and any of its optional aspects as described above.

[0038] In a sixth aspect, an embodiment of the present application provides a computer program. When the computer program runs on a computer, it causes the computer to execute the method according to the first aspect and any of its optional aspects as described above.

[0039] In a seventh aspect, the present application provides a chip system, which includes a processor for supporting a data processing device to implement the functions involved in the above aspects. For example, to send or process the data or information involved in the above method. In a possible design, the chip system further includes a memory for storing the necessary program instructions and data of an execution device or a training device. The chip system may be composed of chips or may include chips and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] To more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the embodiments will be briefly introduced below.

[0041] Figure 1 A schematic diagram of an application architecture provided by an embodiment of the present application;

[0042] Figures 2 to 10 A schematic diagram of an application architecture provided by an embodiment of the present application;

[0043] Figure 11A and Figure 11B A schematic diagram of a CNN;

[0044] Figure 12 A schematic diagram of a data processing method provided by an embodiment of the present application;

[0045] Figures 13 to 15 A schematic diagram of an application architecture provided by an embodiment of the present application;

[0046] Figure 16 and Figure 17 An output schematic diagram of a performance evaluator;

[0047] Figure 18 A processing schematic diagram of a performance evaluator;

[0048] Figure 19 A structural schematic diagram of a data processing device provided by an embodiment of the present application;

[0049] Figure 20 A schematic diagram of a device provided by an embodiment of the present application;

[0050] Figure 21 A schematic diagram of a device provided by an embodiment of the present application;

[0051] Figure 22 A schematic diagram of a chip provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] The embodiments of the present invention will be described below with reference to the accompanying drawings in the embodiments of the present invention. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, rather than intended to limit the present invention.

[0053] The embodiments of the present application will be described below with reference to the accompanying drawings. Those of ordinary skill in the art will know that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

[0054] The terms "first", "second", etc. in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances, which is only a way of distinguishing objects with the same attributes when describing the embodiments of the present application. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product or device comprising a series of units does not have to be limited to those units, but may include other units not clearly listed or inherent to these processes, methods, products or devices.

[0055] First, the overall working process of the artificial intelligence system will be described. Please refer to Figure 1 , Figure 1 shown is a schematic structural diagram of an artificial intelligence main framework. The above artificial intelligence theme framework will be elaborated from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes the condensation process of "data - information - knowledge - wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of artificial intelligence, information (providing and processing technology implementation) to the industrial ecological process of the system.

[0056] (1) Infrastructure

[0057] The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the external world, and is supported through the basic platform. Communicate with the external world through sensors; the computing power is provided by intelligent chips (such as hardware acceleration chips like CPU, NPU, GPU, ASIC, FPGA, etc.); the basic platform includes relevant platform guarantees and supports such as distributed computing frameworks and networks, and may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the external world to obtain data, and these data are provided to the intelligent chips in the distributed computing system provided by the basic platform for computing.

[0058] (2) Data

[0059] The data at the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voices, texts, and also involves the Internet of Things data of traditional devices, including the business data of existing systems and the sensed data such as force, displacement, liquid level, temperature, humidity, etc.

[0060] (3) Data Processing

[0061] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0062] Among them, machine learning and deep learning can perform symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on data.

[0063] Reasoning refers to the process of simulating the intelligent reasoning method of humans in a computer or intelligent system, and using formal information to perform machine thinking and problem-solving according to the reasoning control strategy. The typical function is search and matching.

[0064] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, prediction, etc.

[0065] (4) General Capabilities

[0066] After the data undergoes the above-mentioned data processing, some general capabilities can be formed further based on the results of the data processing. For example, it can be an algorithm or a general system. For example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0067] (5) Intelligent Products and Industry Applications

[0068] Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields, which are the encapsulation of the overall artificial intelligence solution, productize intelligent information decision-making, and realize the landing application. Its application fields mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0069] This application can be but is not limited to being applied in the field of natural language processing in the field of artificial intelligence. Specifically, it can be applied to neural network search in the field of natural language processing and neural network reasoning in the field of natural language processing and other fields. Below, multiple application scenarios landed on products will be introduced.

[0070] To better understand the solution of the embodiments of this application, first, in combination with Figures 2 to 8 A simple introduction to the possible application scenarios of the embodiments of this application will be given below.

[0071] Scenario 1: Neural Network Search

[0072] Reference Figure 2 , this application can be applied to services related to neural network search. Specifically, it can be a neural network architecture search service provided by a cloud-side server. Among them, the user can transmit information related to model search to the neural network search system (such as a cloud server) on the cloud side through a user device. The information related to model search can be the user's performance requirements for the searched model, etc. Then, the server on the cloud side can obtain a search result (such as the target neural network in the embodiments of this application) based on the performance requirements uploaded by the user through a certain neural network search algorithm, and send the search result to the user device.

[0073] Figure 3 FIG. shows a neural network search system 100. The system can obtain training data 102 for training a neural network, validation data 104 for evaluating the performance of the neural network, and performance requirements 103, and use the training data 102, the validation data 104, and the performance requirements 103 to determine a search result 160 (such as the target neural network in the embodiments of this application). The search result 160 is configured to meet the performance requirements 103, that is, receive an input and generate an output that meets the performance requirements 103. The search result 160 can be architecture information of the neural network. The architecture information can define the number of layers of the neural network, the operations performed by each layer, and the connections between the layers in the neural network, that is, which layers receive inputs from other layers in the neural network.

[0074] The system 100 can receive the training data 102, the validation set 104, and the performance requirements 103 in any of various ways. For example, the system 100 can, for example, use an application programming interface (API) available to the system 100 to receive the training data and the performance requirements 103 as an upload from a remote user of the system through a data communication network, and randomly divide the uploaded data into the training data 102 and the validation set 104. As another example, the system 100 can receive an input from the user that specifies which data already maintained by the system 100 should be used to train the neural network, and then divide the specified data into the training data 102 and the validation set 104.

[0075] Generally, the system 100 can determine the search result 160 by searching a space of candidate architectures to identify one or more architectures with the best performance. For example, as Figure 3As shown, the system 100 can search the space of candidate architectures, construct multiple candidate neural network architectures (such as the candidate neural networks in the embodiments of the present application) through the candidate selection engine 130, and perform processing such as model training on the candidate neural network architectures through the training engine 140. The quality evaluation engine 150 can evaluate the training results to determine the search result 160.

[0076] Figure 4 A neural network search system is shown. The neural network search system includes a user device and a neural network search device. Among them, the user device includes intelligent terminals such as mobile phones, personal computers, or information processing centers. The user device is the initiating end of the neural network search. Usually, the user initiates a neural network search request through the user device.

[0077] The above-mentioned neural network search device can be a device or server with neural network search functions such as a cloud server, a network server, an application server, and a management server. The neural network search device receives the neural network search from the intelligent terminal through the interaction interface, and then performs neural network search in ways such as machine learning, deep learning, search, inference, and decision-making through the memory for storing data and the processor link, and feeds back the search result (such as the target neural network in the embodiments of the present application) to the user device. The memory in the neural network search device can be a general term, including local storage and a database for storing historical data. The database can be on the data processing device or on other network servers.

[0078] In Figure 4 the neural network search system shown, the user device can receive the user's instruction. For example, the user device can receive the model performance requirements input by the user for the neural network search, and then initiate a request to the neural network search device.

[0079] In Figure 4 it, the neural network search device can execute the neural network search method of the embodiments of the present application.

[0080] Figure 5A Another neural network search system is shown. In Figure 5A it, the user device directly serves as the neural network search device. The user device can directly receive the model performance requirements input by the user for the neural network search and directly perform neural network search by the hardware of the user device itself. The specific process is similar to Figure 4 the above description and will not be elaborated here.

[0081] In Figure 5A it, the user device itself can execute the neural network search method of the embodiments of the present application.

[0082] I. Image Processing Application Programs

[0083] The product form of the embodiments of this application can be an image processing application program. The image processing application program can run on a terminal device or a server on the cloud side.

[0084] In a possible implementation, the image processing application program can perform image processing tasks and obtain processing results.

[0085] For example, the image processing application program can implement at least image processing tasks based on diffusion methods, but is not limited thereto.

[0086] In a possible implementation, the user can open the image processing application program installed on the terminal device and input image data. The image processing application program can use the model trained by the method provided in the embodiments of this application, or process the image and text by the method provided in the embodiments of this application, and present the processing results to the user (the presentation method can be but is not limited to display, playback, saving, uploading to the cloud side, etc.).

[0087] In a possible implementation, the user can open the image processing application program installed on the terminal device and input image data. The image processing application program can send the image data to the server on the cloud side. The server on the cloud side processes the image or text using the model trained by the method provided in the embodiments of this application, and sends the processing results back to the terminal device. The terminal device can present the processing results to the user (the presentation method can be but is not limited to display, playback, saving, uploading to the cloud side, etc.).

[0088] Exemplarily, the image processing tasks can be specifically applied but are not limited to the following scenarios:

[0089] Scenario 1: Character Recognition.

[0090] Character recognition, as an important AI technology, is widely applied to various financial and legal products. However, with the complexity of character recognition scenarios, the accuracy requirements for character recognition models are getting higher and higher. Since the Transformer network has long-distance context dependencies and can better capture the global features of text, it has gradually been applied to character recognition models to improve the accuracy of character recognition. The Transformer network requires a large amount of labeled data, and synthetic datasets are usually used for pre-training to reduce the labeling cost, but it will introduce domain gaps. It is hoped to use more generalized pre-training algorithms to improve the generalization ability of pre-trained models and ultimately improve the tuning accuracy on business data.

[0091] Scenario 2: Image Classification

[0092] Image classification refers to the process of dividing different images into different category labels. From the perspective of a computer, an image is a matrix with values ranging from 0 to 255. The computer analyzes the matrix to obtain the category result, which is image classification in computer vision.

[0093] Image classification is the most fundamental problem in the field of computer vision. The problem it solves is: given an image, correctly give the category to which the image belongs. It is the most basic and fundamental task, often used as a preprocessing for other tasks or integrated with other tasks. Object detection is also a subtask of image classification.

[0094] Next, the image processing application in the embodiments of this application will be introduced respectively from the functional architecture and the product architecture for implementing the functions.

[0095] Refer to Figure 5B , Figure 5B which is a schematic diagram of the functional architecture of the image processing application in the embodiments of this application:

[0096] In a possible implementation, as shown in Figure 5B , the image processing application 102 can receive the input parameter 101 (such as an image or text) and generate a processing result 103. The image processing application 102 can be executed on (for example) at least one computer system and includes computer code that, when executed by one or more computers, causes the computer to execute a model trained by the method provided by the embodiments of this application.

[0097] Refer to Figure 6 , Figure 6 which is a schematic diagram of the entity architecture for running the image processing application in the embodiments of this application:

[0098] See Figure 6 , Figure 6 which shows a schematic diagram of a system architecture. The system may include a terminal 100 and a server 200. Among them, the server 200 may include one or more servers ( Figure 6 illustrated by taking one server as an example), and the server 200 can provide image processing functions for one or more terminals.

[0099] Among them, an image processing application may be installed on the terminal 100, or a web page related to the image processing function may be opened. The above application and web page can provide an interface. The terminal 100 can receive the relevant parameters input by the user on the image processing function interface and send the above parameters to the server 200. The server 200 can obtain a processing result based on the received parameters and return the processing result to the terminal 100.

[0100] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself, without the cooperation of the server. The embodiments of the present application do not limit this.

[0101] Next, the product form of the terminal 100 will be described. Figure 6 in the terminal 100;

[0102] The terminal 100 in the embodiments of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiments of the present application do not impose any restrictions on this.

[0103] Figure 7 FIG. shows an optional schematic hardware structure of the terminal 100.

[0104] Referring to Figure 7 as shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a processor 170, an external interface 180, a power supply 190, and other components. Those skilled in the art can understand that Figure 7 This is merely an example of a terminal or a multifunctional device, and does not constitute a limitation on the terminal or the multifunctional device. It may include more or fewer components than shown in the figure, or combine certain components, or different components.

[0105] The input unit 130 can be used to receive input digital or character information, and generate key signal inputs related to the user settings and function controls of the portable multifunctional device. Specifically, the input unit 130 can include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect touch operations of the user thereon or nearby (such as operations of the user using any suitable object such as a finger, a joint, a stylus, etc. on or near the touch screen), and drive the corresponding connection device according to a preset program. The touch screen can detect the touch action of the user on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the commands sent by the processor 170; the touch signal at least includes contact coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, multiple types such as resistive, capacitive, infrared, and surface acoustic wave can be used to implement the touch screen. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, the other input devices 132 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, etc.

[0106] Among them, the other input devices 132 can receive input image data.

[0107] The display unit 140 can be used to display information input by the user or information provided to the user, various menus of the terminal 100, an interactive interface, file display, and / or the playback of any multimedia file. In the embodiment of the present application, the display unit 140 can be used to display the interface of an image processing application program, processing results, etc.

[0108] The memory 120 can be used to store instructions and data. The memory 120 mainly includes a storage instruction area and a storage data area. The storage data area can store various data, such as multimedia files, texts, etc.; the storage instruction area can store software units such as an operating system, applications, instructions required for at least one function, etc., or their subsets or extended sets. It can also include a non-volatile random access memory; it provides the processor 170 with functions including managing the hardware, software, and data resources in the computing processing device, supporting control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0109] The processor 170 is the control center of the terminal 100, connecting various parts of the entire terminal 100 through various interfaces and circuits. By running or executing the instructions stored in the memory 120 and invoking the data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby exercising overall control over the terminal device. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 170 either. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be separately implemented on independent chips. The processor 170 can also be used to generate corresponding operation control signals, send them to the corresponding components of the computing and processing device, read and process the data in the software, especially read and process the data and programs in the memory 120, so that each functional module therein executes the corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0110] Among them, the memory 120 can be used to store software codes related to the data processing method. The processor 170 can execute the steps of the data processing method of the chip, or can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to implement the corresponding functions.

[0111] The radio frequency unit 110 (optional) can be used for receiving and transmitting information or signals during a call. For example, after receiving the downlink information from the base station, it is sent to the processor 170 for processing; in addition, the uplink data designed is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. This wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0112] Wherein, in the embodiment of the present application, the radio frequency unit 110 can send image data to the server 200 and receive the processing result sent by the server 200.

[0113] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network interface.

[0114] The terminal 100 further includes a power supply 190 (such as a battery) for powering each component. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system.

[0115] The terminal 100 further includes an external interface 180. The external interface can be a standard Micro USB interface or a multi-pin connector, which can be used to connect the terminal 100 to other devices for communication and can also be used to connect a charger to charge the terminal 100.

[0116] Although not shown, the terminal 100 may further include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be elaborated here. Some or all of the methods described below can be applied to the terminal 100 as Figure 7 shown.

[0117] Next, the product form of the server 200 will be described. Figure 6 in the server 200;

[0118] Figure 8 A schematic structural diagram of a server 200 is provided, as Figure 8 shown. The server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other through the bus 201.

[0119] The bus 201 can be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 8 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0120] The processor 202 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0121] The memory 204 can include volatile memory, such as random access memory (RAM). The memory 204 can also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0122] Among them, the memory 204 can be used to store software codes related to data processing methods. The processor 202 can execute the steps of the data processing method of the chip or schedule other units to implement corresponding functions.

[0123] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with the function of executing instructions, such as CPU, DSP, etc., or a hardware system without the function of executing instructions, such as ASIC, FPGA, etc., or a combination of the above-mentioned hardware system without the function of executing instructions and the hardware system with the function of executing instructions.

[0124] It should be understood that the steps related to the model inference process in the embodiments of the present application involve AI-related operations. When performing AI operations, the instruction execution architectures of the terminal device and the server are not limited to the architecture of the above-mentioned processor combined with the memory. The following will be combined with Figure 9 to introduce the system architecture provided by the embodiments of the present application in detail.

[0125] Figure 9 is a schematic diagram of the system architecture provided by the embodiments of the present application. As Figure 9 shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0126] The execution device 510 includes a computing module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The computing module 511 may include a target model / rule 501, and the preprocessing module 513 and the preprocessing module 514 are optional.

[0127] Among them, the execution device 510 can be the above-mentioned terminal device or server that runs image processing applications.

[0128] The data acquisition device 560 is used to acquire training samples. The training samples can be image data, etc. After acquiring the training samples, the data acquisition device 560 stores these training samples in the database 530.

[0129] The training device 520 can maintain the training samples in the database 530 and train the neural network to be trained (such as the neural network model in the embodiments of the present application) to obtain the target model / rule 501.

[0130] It should be understood that the training device 520 can pre-train the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model on the basis of pre-training.

[0131] It should be noted that in practical applications, the training samples maintained in the database 530 do not necessarily all come from the acquisition of the data acquisition device 560, and it is also possible to receive them from other devices. Additionally, it should be noted that the training device 520 does not necessarily train the target model / rule 501 entirely based on the training samples maintained in the database 530, and it is also possible to obtain training samples from the cloud or other places for model training. The above description should not be construed as a limitation on the embodiments of the present application.

[0132] The target model / rule 501 trained according to the training device 520 can be applied to different systems or devices, such as being applied to Figure 9 the execution device 510 shown in the figure. The execution device 510 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle-mounted terminal, etc., or it can also be a server, etc.

[0133] Specifically, the training device 520 can transfer the trained model to the execution device 510.

[0134] In Figure 9 , the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with external devices. The user can input data (such as image data in the embodiments of the present application) to the I / O interface 512 through the client device 540.

[0135] The preprocessing modules 513 and 514 are used to preprocess the input data received according to the I / O interface 512. It should be understood that there may be no preprocessing modules 513 and 514 or only one preprocessing module. When the preprocessing modules 513 and 514 do not exist, the computing module 511 can directly process the input data.

[0136] During the preprocessing of the input data by the execution device 510, or during the relevant processing such as the computing module 511 of the execution device 510 performing calculations, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, or can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 550.

[0137] Finally, the I / O interface 512 provides the processing result to the client device 540 and thus to the user.

[0138] In Figure 9 the illustrated case, the user can manually give input data, and the "manually given input data" can be operated through the interface provided by the I / O interface 512. In another case, the client device 540 can automatically send input data to the I / O interface 512. If the client device 540 is required to automatically send input data and user authorization is needed, the user can set corresponding permissions in the client device 540. The user can view the result output by the execution device 510 in the client device 540, and the specific presentation form can be specific ways such as display, sound, action, etc. The client device 540 can also be used as a data acquisition end to acquire the input data input to the I / O interface 512 and the output result of the output I / O interface 512 as shown in the figure as new sample data and store it in the database 530. Of course, it can also be acquired without going through the client device 540, but directly by the I / O interface 512 taking the input data input to the I / O interface 512 and the output result of the output I / O interface 512 as shown in the figure as new sample data and storing it in the database 530.

[0139] It should be noted that Figure 9 is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 9 the data storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the above execution device 510 can be deployed in the client device 540.

[0140] Next, a more detailed architecture of the execution entity for executing the neural network search method in the embodiment of the present application will be introduced.

[0141] Next, Figure 9 a detailed introduction to the system architecture provided by the embodiment of the present application will be given in conjunction with Figure 9 is a schematic diagram of the system architecture provided by the embodiment of the present application. As Figure 9 shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0142] The execution device 510 includes a computing module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The computing module 511 may include a target model / rule 501, and the preprocessing module 513 and the preprocessing module 514 are optional.

[0143] The data acquisition device 560 is used to acquire training samples. The training samples can be image data, text data, audio data, etc. In the embodiments of the present application, the training samples are the data used for training multiple candidate neural networks. After the training samples are acquired, the data acquisition device 560 stores these training samples in the database 530.

[0144] It should be understood that a search space can also be maintained in the database 530.

[0145] The training device 520 can construct multiple candidate neural networks based on the search space maintained in the database 530, and train the multiple candidate neural networks based on the training samples to search for the target model / rule 501. In the embodiments of the present application, the target model / rule 501 can be a target neural network.

[0146] It should be noted that in practical applications, the training samples maintained in the database 530 do not necessarily all come from the acquisition of the data acquisition device 560, and it is also possible to receive them from other devices. Additionally, it should be noted that the training device 520 does not necessarily train the target model / rule 501 completely based on the training samples maintained in the database 530, and it is also possible to obtain training samples from the cloud or other places for model training. The above description should not be used as a limitation to the embodiments of the present application.

[0147] The target model / rule 501 trained according to the training device 520 can be applied to different systems or devices, such as applied to Figure 9 the execution device 510 shown. The execution device 510 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle-mounted terminal, etc., and can also be a server or the cloud, etc.

[0148] Specifically, the training device 520 can transfer the target neural network to the execution device 510.

[0149] In Figure 9 it, the execution device 510 configures an input / output (I / O) interface 512 for data interaction with external devices. The user can input data (such as the data to be processed in the embodiments of the present application) to the I / O interface 512 through the client device 540.

[0150] The preprocessing modules 513 and 514 are used to perform preprocessing on the input data received by the I / O interface 512. It should be understood that there may be no preprocessing modules 513 and 514 or only one preprocessing module. When the preprocessing modules 513 and 514 do not exist, the computing module 511 can directly process the input data.

[0151] When the execution device 510 preprocesses the input data or the computing module 511 of the execution device 510 performs related processing such as computing, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0152] Finally, the I / O interface 512 presents the processing result (such as the data processing result in the embodiments of the present application) to the client device 540, so as to provide it to the user.

[0153] From the inference side of the model:

[0154] In the embodiments of the present application, the computing module 511 of the above execution device 520 can obtain the code stored in the data storage system 550 to implement the data processing method in the embodiments of the present application.

[0155] In the embodiments of the present application, the computing module 511 of the execution device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, a DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, an FPGA, etc., or a combination of the above hardware systems without an instruction execution function and a hardware system with an instruction execution function.

[0156] Specifically, the computing module 511 of the execution device 520 may be a hardware system with an instruction execution function. The data processing method provided in the embodiments of the present application may be software code stored in a memory. The computing module 511 of the execution device 520 can obtain the software code from the memory and execute the obtained software code to implement the data processing method provided in the embodiments of the present application.

[0157] It should be understood that the computing module 511 of the execution device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps of the data processing method provided in the embodiments of the present application can also be implemented by the hardware system without the function of executing instructions in the computing module 511 of the execution device 520, which is not limited herein.

[0158] From the perspective of the training side of the model:

[0159] In the embodiments of the present application, the above-mentioned training device 520 can obtain the code stored in a memory ( Figure 9 not shown in the figure, which can be integrated with the training device 520 or deployed separately from the training device 520) to implement the neural network search method in the embodiments of the present application.

[0160] In the embodiments of the present application, the training device 520 can include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors, or microcontrollers, etc.), or a combination of these hardware circuits. For example, the training device 520 can be a hardware system with the function of executing instructions, such as a CPU, a DSP, etc., or a hardware system without the function of executing instructions, such as an ASIC, an FPGA, etc., or a combination of the above-mentioned hardware systems without the function of executing instructions and a hardware system with the function of executing instructions.

[0161] Specifically, the training device 520 can be a hardware system with the function of executing instructions. The data processing method provided in the embodiments of the present application can be software code stored in a memory. The training device 520 can obtain the software code from the memory and execute the obtained software code to implement the neural network search method provided in the embodiments of the present application.

[0162] It should be understood that the training device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps of the neural network search method provided in the embodiments of the present application can also be implemented by the hardware system without the function of executing instructions in the training device 520, which is not limited herein.

[0163] II. Cloud services of image processing function types provided by the server:

[0164] In a possible implementation, the server can provide services for image processing functions to the terminal side through an application programming interface (API).

[0165] Among them, the terminal device can send relevant parameters (such as data like images) to the server through the API provided by the cloud. The server can obtain processing results, etc., based on the received parameters, and return the processing results to the terminal.

[0166] The descriptions of the terminal and the server can be referred to the descriptions in the above embodiments, and will not be elaborated here.

[0167] As Figure 10 shows the process of using a cloud service of image processing functions provided by a cloud platform.

[0168] 1. Open and purchase the image processing service.

[0169] 2. The user can download the software development kit (SDK) corresponding to the image processing service. Usually, the cloud platform provides multiple development versions of the SDK for the user to choose according to the requirements of the development environment, such as the SDK of the JAVA version, the SDK of the python version, the SDK of the PHP version, the SDK of the Android version, etc.

[0170] 3. After the user downloads the corresponding version of the SDK to the local according to the requirements, import the SDK project into the local development environment, configure and debug it in the local development environment. The local development environment can also develop other functions, so as to form an application integrating the capabilities of image processing functions.

[0171] 4. During the use of the image processing function class application, when the image processing function is required, the API call of the image processing function can be triggered. When the application triggers the image processing function, an API request is sent to the running instance of the image processing function class service in the cloud environment. Among them, the API request carries an image or text, and the running instance in the cloud environment processes the image to obtain the processing result.

[0172] 5. The cloud environment returns the processing result to the application, thus completing a call of the image processing function.

[0173] Since the embodiments of the present application involve a large number of applications of neural networks, for the sake of easy understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application will be introduced below.

[0174] (1) Neural network

[0175] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs and an intercept of 1 as inputs. The output of this operation unit can be:

[0176]

[0177] where s = 1, 2, …, n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting multiple such single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be a region composed of several neural units.

[0178] (2) A convolutional neural network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network contains a feature extractor composed of a convolutional layer and a subsampling layer, and this feature extractor can be regarded as a filter. The convolutional layer refers to the neuron layer in the convolutional neural network that performs convolutional processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neighboring layer neurons. In a convolutional layer, there are usually several feature planes, and each feature plane can be composed of some neurals units arranged in a rectangle. The neural units in the same feature plane share weights, and the shared weight here is the convolutional kernel. Sharing weights can be understood as a way of extracting features that is independent of position. The convolutional kernel can be formalized in the form of a matrix with a random size, and the convolutional kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit brought by sharing weights is to reduce the connections between the layers of the convolutional neural network and at the same time reduce the risk of overfitting.

[0179] CNN is a very common neural network. As introduced in the basic concepts above, the convolutional neural network is a deep neural network with a convolutional structure and is a deep learning architecture. A deep learning architecture refers to performing multiple levels of learning at different abstraction levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, and each neuron in this feed-forward artificial neural network can respond to the input.

[0180] The convolutional neural network (CNN) 200 may include an input layer 210, a convolutional / pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230.

[0181] Convolutional / pooling layer 220:

[0182] Convolutional layer:

[0183] As Figure 11A shown, the convolutional / pooling layer 220 may include layers such as examples 221 - 226. For example: in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, 225 is a convolutional layer, and 226 is a pooling layer; in another implementation, 221 and 222 are convolutional layers, 223 is a pooling layer, 224 and 225 are convolutional layers, and 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input of the subsequent pooling layer or as the input of another convolutional layer to continue the convolution operation.

[0184] Next, taking the convolutional layer 221 as an example, the internal working principle of a single convolutional layer will be introduced.

[0185] The convolutional layer 221 may include a number of convolutional operators, also known as kernels, which act as filters for extracting specific information from the input image matrix in image processing. Essentially, a convolutional operator can be a weight matrix, which is usually predefined. Taking an image as an example (similar for other data types), during the process of performing a convolution operation on the image, the weight matrix typically processes the input image pixel by pixel (or two pixels at a time... depending on the value of the stride) along the horizontal direction, thus completing the work of extracting specific features from the image. The size of this weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as that of the input image, and during the convolution operation, the weight matrix extends to the entire depth of the input image. Therefore, convolving with a single weight matrix will produce a convolved output with a single depth dimension. However, in most cases, instead of using a single weight matrix, multiple weight matrices with the same size (rows × columns), i.e., multiple matrices of the same type, are applied. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image, and here the dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix is used to extract edge information of the image, another weight matrix is used to extract specific colors of the image, and yet another weight matrix is used to blur the unwanted noise in the image, etc. These multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size also have the same size. Then, the multiple feature maps of the same size that are extracted are combined to form the output of the convolution operation.

[0186] The weight values in these weight matrices need to be obtained through a large amount of training in practical applications. Each weight matrix formed by the weight values obtained through training can be used to extract information from the input image, enabling the convolutional neural network 200 to make correct predictions.

[0187] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (such as 221) often extracts more general features, which can also be called low-level features; as the depth of the convolutional neural network 200 increases, the later convolutional layers (such as 226) extract increasingly complex features, such as high-level semantic features. The higher the semantic features, the more suitable they are for the problem to be solved.

[0188] Pooling layer:

[0189] Since it is often necessary to reduce the number of training parameters, a pooling layer is often introduced periodically after the convolutional layer. Figure 11AFor each of the layers 221 - 226 shown in 220, it can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. During the image processing, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a max pooling operator for sampling the input image to obtain a smaller-sized image. The average pooling operator can calculate the average value of the pixel values in the image within a specific range as the result of average pooling. The max pooling operator can take the pixel with the maximum value within the specific range as the result of max pooling. Additionally, just as the size of the weight matrix in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after being processed by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel point in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

[0190] Fully connected layer 230:

[0191] After being processed by the convolutional layer / pooling layer 220, the convolutional neural network 200 is still not sufficient to output the required output information. Because as mentioned before, the convolutional layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to use the fully connected layer 230 to generate one or a set of outputs with the number of classes required. Therefore, the fully connected layer 230 can include multiple hidden layers (such as Figure 11A 231, 232 to 23n shown), and the parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type can include image recognition, image classification, image super-resolution reconstruction, and so on...

[0192] After the multiple hidden layers in the fully connected layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240. The output layer 240 has a loss function similar to categorical cross-entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 11A the propagation from 210 to 240 is the forward propagation) is completed, the backpropagation (such as Figure 11A the propagation from 240 to 210 is the backpropagation) will start to update the weight values and biases of the previously mentioned layers to reduce the loss of the convolutional neural network 200, that is, the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0193] It should be noted that as Figure 11AThe convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models. For example, it can only include Figure 11A a part of the network structure shown in Figure 11A . For example, the convolutional neural network adopted in the embodiments of the present application can only include an input layer 210, a convolutional layer / pooling layer 220, and an output layer 240.

[0194] It should be noted that, as Figure 11A the convolutional neural network 100 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models. For example, as Figure 11B shown, multiple convolutional layers / pooling layers are in parallel, and the features extracted separately are all input to a fully connected layer 230 for processing.

[0195] (6) Loss function

[0196] During the process of training a deep neural network, since it is desired that the output of the deep neural network is as close as possible to the value that is truly desired to be predicted, the predicted value of the current network and the truly desired target value can be compared, and then the weight vector of each layer of the neural network can be updated according to the difference between the two (of course, there is usually an initialization process before the first update, that is, parameters are pre-configured for each layer in the deep neural network). For example, if the predicted value of the network is high, the weight vector is adjusted to make it predict lower, and continuous adjustment is made until the deep neural network can predict the truly desired target value or a value very close to the truly desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function. They are important equations for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then, the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0197] (7) Backpropagation algorithm

[0198] The convolutional neural network can use the backpropagation (BP) algorithm to correct the magnitude of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the input signal is propagated forward until the output generates an error loss, and the error loss information is propagated backward to update the parameters in the initial super-resolution model, so as to converge the error loss. The backpropagation algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the parameters of the optimal super-resolution model, such as the weight matrix.

[0199] (8) Memristor: A memristor, also known as a memory resistor, is a passive electronic component. Similar to a resistor, a memristor can generate and maintain a safe current through a device. However, what differentiates it from a resistor is that a memristor can "remember" the amount of charge that passed through it previously even after the power is turned off.

[0200] Refer to Figure 12 , Figure 12 which is a schematic illustration of an embodiment of a data processing method provided by an embodiment of this application. As Figure 12 shown, the data processing method provided by the embodiments of this application may include:

[0201] 1201. Based on the input data of the convolutional layer, obtain a mask of the input data through a prediction network; the input data includes data in the channel dimension and the spatial dimension; the mask includes a mask in the channel dimension and a mask in the spatial dimension.

[0202] In a possible implementation, Figure 12 the corresponding embodiment may be a step in the feedforward process during model training or a step in the model inference process. Refer to Figure 13 , Figure 13 which is a schematic illustration of the application architecture of this application.

[0203] Specifically, it may be a process of processing input data (such as an image) through a target network (the target network is a machine learning model), where the target network may include multiple network layers, and the multiple network layers may include a convolutional layer.

[0204] For example, the convolutional layer may be an intermediate layer of the target network. The target network may further include a first network connected before the convolutional layer and a second network connected after the convolutional layer. Furthermore, when the target network processes the input data, the result obtained by the first network (such as a multi-channel feature map) may be used as the input of the convolutional layer, the output of the convolutional layer may be used as the input of the second network, and the output of the second network may be used as the output of the target network.

[0205] In a possible implementation, the target network is used to process the image data for a target task to obtain the processing result. The target task is one of image classification, target recognition, text recognition in an image, image segmentation, image enhancement, or other image processing tasks, which is not limited in this application.

[0206] To improve the fault tolerance of a machine learning model, fault tolerance training can be added to the training of the dynamic inference module. The so-called fault tolerance can be understood as: when the hardware processes data through the model, when part of the output result of the intermediate layer of the network is missing due to hardware reasons (that is, read / write noise is generated during hardware operation), the model can still have a high processing accuracy. During training, simulated random noise is injected into the model to approximate the read / write noise generated during actual hardware operation, so that the mask generator for dynamic inference can adapt to the corresponding intensity of read / write noise to a certain extent and give the optimal mask. During the inference stage, the spatial mask generator and the channel mask generator respectively give the corresponding masks, which act on the output of the network layer (such as the feature map) and the weights of the model.

[0207] Among them, when the mask acts on the output of the intermediate layer (that is, the input of the next network layer), it is necessary to calculate the prefix sum through hardware based on the mask. Through the prefix sum, the mask can act on the data, and then the network layer can process the masked data.

[0208] The data can include data in the channel dimension and data in the spatial dimension. The mask can include masks in the channel dimension and the spatial dimension. When the convolutional layer performs a convolutional operation, it can perform a sliding window convolutional operation on the input data through the convolutional kernel. In the prior art, each time the sliding window is performed, it is necessary to calculate the prefix sum through the mask (including the prefix sum in the channel dimension and the prefix sum in the spatial dimension). However, calculating the prefix sum needs to be implemented through hardware units. The prefix sum including the prefix sum in the channel dimension and the prefix sum in the spatial dimension is three-dimensional data (that is, a large amount of data). Therefore, a large amount of computing resources are required, and a large computing delay will be caused.

[0209] This application is to solve the above problems. Specifically, for the same input data (such as the same picture), the channel mask is constant, and only the spatial mask changes with the sliding window. Therefore, the channel mask can be preprocessed to calculate the prefix sum in the channel dimension in advance. Therefore, each time the sliding window is performed, only the prefix sum in the spatial dimension needs to be calculated, and the mask is applied to the input data using the fusion result of the prefix sum in the channel dimension and the prefix sum in the spatial dimension. Equivalently, when the convolutional layer performs each sliding window for the same input data, it only needs to reuse the pre-calculated prefix sum in the channel dimension and only needs to recalculate the prefix sum in the spatial dimension, greatly reducing the demand for computing resources and computing delay.

[0210] 1202. Obtain the prefix sum in the channel dimension according to the mask in the channel dimension.

[0211] 1203. Obtain the prefix sum in the spatial dimension when performing a single sliding window convolution on the input data by the convolutional kernel of the convolutional layer according to the mask in the spatial dimension.

[0212] In a possible implementation, the prefix sum in the channel dimension can be obtained according to the mask in the channel dimension.

[0213] In a possible implementation, the prefix sum in the spatial dimension can be obtained according to the mask in the spatial dimension. For example, the prefix sum in the spatial dimension can be obtained according to the mask of the convolutional data object in the spatial dimension each time a single sliding window convolution is performed.

[0214] In a possible implementation, the prefix sum in the channel dimension can be obtained through a first hardware unit according to the mask in the channel dimension;

[0215] In a possible implementation, the prefix sum in the channel dimension can be obtained through a second hardware unit according to the mask in the spatial dimension, where the first hardware unit and the second hardware unit are different.

[0216] For example, reference can be made to Figure 14 , Figure 14 As a schematic illustration of the implementation of a dynamic inference hardware architecture, in the embodiments of the present application, the main design goal of the hardware architecture is to non - continuously activate the RRAM rows with mask values of 1 for the mask acting on the input, so as to dynamically select some RRAM cells for calculation. The embodiments of the present application overcome the shortcomings of existing work in the implementation of the hardware architecture. For example, a mask of length N is decomposed into the Cartesian product of a spatial mask and a channel mask, and the two are calculated separately and then combined. For the same picture, the channel mask is constant, and only the spatial mask changes with the sliding window. Therefore, the channel mask can be pre - processed to calculate its prefix sum in advance. Therefore, after each sliding window, we only need to calculate the prefix sum of the spatial mask, greatly reducing the demand for computing resources and computing latency.

[0217] Through the above - mentioned method, the model structure can be dynamically adjusted during the inference process, and through the design of the hardware architecture, the efficiency of dynamic mask selection is improved, and a significant improvement in the fault - tolerance performance under similar throughput is achieved.

[0218] 1204. According to the fusion result of the prefix sum in the channel dimension and the prefix sum in the spatial dimension, apply the mask to the input data, and perform the sliding window convolution processing on the masked input data through the convolutional kernel of the convolutional layer.

[0219] In a possible implementation, the fusion result is the Cartesian product of the prefix sum in the channel dimension and the prefix sum in the spatial dimension. That is, the Cartesian product can be performed on the prefix sum in the channel dimension and the prefix sum in the spatial dimension to obtain the fusion result, and the fusion result can apply the mask to the input data and perform the sliding window convolution process on the masked input data through the convolution kernel of the convolutional layer.

[0220] In a possible implementation, the third hardware unit can be used to fuse the prefix sum in the channel dimension and the prefix sum in the spatial dimension to obtain the fusion result, and the third hardware unit is different from the first hardware unit and the second hardware unit.

[0221] During the process of neural network search, multiple candidate neural networks can be constructed through search. When constructing a candidate neural network, the types of each network layer included in the candidate neural network can be determined based on a sampling method, and then the network layer can be sampled to complete the construction of the candidate neural network. After obtaining multiple candidate neural networks, the performance of each candidate neural network can be calculated by a performance evaluator (such as a neural network), and then based on the performance of each candidate neural network, a network with better performance (such as the target network in the embodiments of the present application) can be selected from the multiple candidate neural networks. For example, the target network can be the model with the best performance among the multiple candidate neural networks.

[0222] When fault-tolerant training is applied to neural network search, the performance needs to represent the fault-tolerant performance of the model. However, there is a large degree of randomness in the fault-tolerant performance test. The randomness mainly comes from the following three sources:

[0223] 1. Randomness of fault-tolerant training: The weights obtained by training the same model with different random seeds are different, and there must be differences in its fault-tolerant performance; 2. Randomness of error injection testing: For a model, even with the same intensity but different errors, it will have different performances; 3. Randomness during actual deployment: The specific errors generated when the model is actually deployed on different hardware are also different.

[0224] The randomness causes the performance gap observed in a single instance not to fully indicate the excellent performance of the two architectures. First, the result obtained by simply deciding the performance ranking through a single observation is likely to be one-sided. Take Figure 16Taking the shown situation as an example, architecture A is inferior to architecture B in Observation 1, but the opposite is true in Observation 2. Second, taking the average of multiple tests can obtain the average performance ranking, but there is no knowledge about information such as its upper bound and lower bound. The performance obtained during actual model deployment can be regarded as a single sampling from the performance distribution and cannot be described only by the mean. In particular, a single evaluation of the fault-tolerant performance can be regarded as a sampling with accidental uncertainty from the distribution. For example, assume that there is currently a fault-tolerant performance distribution that follows a Gaussian distribution, and the sampling points closer to the mean have a greater probability of being sampled, that is, the uncertainty is smaller. It is hoped that the predictor can model the overall fault-tolerant performance distribution corresponding to the input architecture. Figure 17 It vividly shows the difference between the performance evaluator and the traditional predictor in the embodiments of the present application.

[0225] That is to say, in a possible implementation, the method is applied to the data processing process of the target network; the target network is obtained through neural network search; multiple candidate neural networks are obtained; the target network is selected from the multiple candidate neural networks according to the performance of each candidate neural network; the performance is represented by the mean of the performance distribution and the standard deviation of the performance distribution.

[0226] Among them, the performance evaluator may include a Bayesian model and an MLP, referring to Figure 18 , Figure 18 which is a structural schematic of the performance evaluator.

[0227] In a possible implementation, the mean and the standard deviation are determined through Bayesian inference and Monte Carlo sampling.

[0228] In the embodiments of the present application, the Monte Carlo random inactivation method can be applied to the predictor, and each forward inference is equivalent to a single sampling of the fault-tolerant performance of the input architecture. For this input architecture, when the number of inferences is large enough, the sampled performance values output show the target fault-tolerant performance distribution. An additional prediction head can be used to map the architecture hidden representation to the "standard deviation" of the fault-tolerant performance evaluation of this architecture. The larger this value is, the higher the degree of diffusion. In particular, this standard deviation does not directly correspond to the standard deviation of the true performance distribution, but is only a measure that can be learned by the model.

[0229]

[0230]

[0231] For the design of the loss function, a certain architecture α to be evaluated can be considered. The training data includes the fault tolerance performance of its independent sampling D times, and the performance obtained from the i-th sampling is denoted as y_i. During the training process, the predictor performs forward inference D times to obtain D predicted performance values. The loss function is as follows. This formula is derived from the Gaussian distribution, where s is the "log variance". In particular, an additional regularization term can be added to encourage the model to learn the distribution spread by minimizing the L2 distance between the sampled minimum (maximum) value and the training data minimum (maximum) value.

[0232] An embodiment of the present application designs a dynamic inference method and a hardware architecture adapted to fault tolerance training to select a better fault tolerance inference path through dynamic inference on computing platforms such as memristors and improve the model fault tolerance performance. At the same time, a performance predictor based on the Bayesian model-Monte Carlo sampling method is designed to predict both the mean and standard deviation of the model performance simultaneously to obtain a model with a higher mean and more stable performance in the presence of uncertain noise.

[0233] In the embodiment of the present application, through the design of a performance predictor for fault tolerance architecture search, the mean of the model performance can be better predicted, and the standard deviation can also be effectively predicted, so that the obtained model has both a good average level and can perform relatively stably.

[0234] Refer to Figure 15 , in the embodiment of the present application, in the model design stage, a better architecture search can be performed using a performance predictor that can completely model the performance distribution to obtain the model architecture. Subsequently, fault tolerance training with a dynamic inference module is performed on this architecture, and then the model is deployed to the hardware architecture designed by us that is adapted to the dynamic inference module. During inference, the dynamic inference module generates a channel mask and a spatial mask according to the output image, and the corresponding module in the hardware architecture can quickly complete the dynamic selection of the results using sparsity, and perform convolution calculations on the weights and feature maps not masked by the mask to obtain the final result.

[0235] Next, the beneficial effects of the embodiment of the present application will be introduced in combination with the experimental results:

[0236] The dynamic inference method of the embodiment of the present application was experimentally tested on the resnet-20 classification model. Referring to Table 1, it can be seen that the method of the embodiment of the present application has a significant improvement in fault tolerance ability in the case of more significant errors.

[0237] Table 1

[0238]

[0239]

[0240] In addition, a test set for architecture search is also constructed, and 400 architectures are sampled using the NDS-resnet search space to evaluate the performance evaluator of this application.

[0241] Table 2

[0242] 0.05 0.1 0.25 0.5 0.75 1.0 mean 0.1930±0.0502 0.3755±0.0411 0.4420±0.0380 0.6167±0.0178 0.7104±0.0107 0.7297±0.0036 std 0.0992±0.0346 0.1854±0.0425 0.1467±0.0417 0.1866±0.0558 0.4860±0.0138 0.4878±0.0182 min 0.1525±0.0388 0.3161±0.0360 0.3566±0.0427 0.522±0.0221 0.6349±0.0118 0.6197±0.0211 max 0.2584±0.0454 0.4058±0.0402 0.4824±0.0393 0.5935±0.0123 0.6347±0.0184 0.6522±0.0138

[0243] Kendall correlation between prediction and sampled values (naive predictor)

[0244] 0.05 0.1 0.25 0.5 0.75 1.0 mean 0.2162±0.0409 0.3678±0.0641 0.5750±0.0617 0.7746±0.0102 0.8331±0.0044 0.8291±0.0086 std 0.1847±0.0643 0.3088±0.0527 0.5035±0.0568 0.7101±0.0087 0.7566±0.0090 0.7547±0.0103 min 0.2129±0.0334 0.3357±0.0465 0.5168±0.0559 0.6946±0.0137 0.7487±0.0058 0.7404±0.0103 max 0.2177±0.0414 0.3967±0.0664 0.5819±0.0551 0.7469±0.0132 0.7850±0.0062 0.7835±0.0091

[0245] Kendall correlation between prediction and sampled values (fault-tolerant predictor)

[0246] Referring to Table 2, it can be seen that the method of the embodiment of this application has a considerable improvement in mean prediction, and the improvement in the prediction of standard deviation and extreme values is very significant.

[0247] Referring to Figure 19 , Figure 19 is a schematic diagram of an embodiment of a data processing device provided by an embodiment of this application. As Figure 19 shown, the data processing device 1900 provided by the embodiment of this application may include:

[0248] A processing module 1901, configured to obtain a mask of the input data through a prediction network according to the input data of the convolutional layer; the input data includes data in the channel dimension and the spatial dimension; the mask includes a mask in the channel dimension and a mask in the spatial dimension;

[0249] For the description of the processing module 1901, reference may be made to the descriptions of steps 1201 and 1204 in the above embodiments, which will not be repeated here.

[0250] A first hardware unit 1902, configured to obtain a prefix sum in the channel dimension according to the mask in the channel dimension;

[0251] For the description of the first hardware unit 1902, reference may be made to the description of step 1202 in the above embodiments, which will not be repeated here.

[0252] A second hardware unit 1903, configured to obtain a prefix sum in the spatial dimension when performing a single sliding window convolution on the input data by the convolution kernel of the convolutional layer according to the mask in the spatial dimension;

[0253] For the description of the second hardware unit 1903, reference may be made to the description of step 1203 in the above embodiments, which will not be repeated here.

[0254] The processing module 1901 is further configured to actuate the mask on the input data according to the fusion result of the prefix sum in the channel dimension and the prefix sum in the spatial dimension, and perform the sliding window convolution processing on the masked input data through the convolution kernel of the convolutional layer.

[0255] In a possible implementation, the fusion result is the Cartesian product of the prefix sum in the channel dimension and the prefix sum in the spatial dimension.

[0256] In a possible implementation, the input data is a multi-channel feature map.

[0257] In a possible implementation, the processing module is further configured to:

[0258] Process the image data through a first network to obtain the input data of the convolutional layer;

[0259] Process the output data of the convolutional layer through a second network to obtain the processing result of the image data. The first network, the second network, and the convolutional layer belong to a target network, and the target network is used to process the image data for a target task to obtain the processing result, and the target task is one of image classification, target recognition, text recognition in an image, image segmentation, and image enhancement.

[0260] In a possible implementation, the apparatus is applied to the data processing process of the target network;

[0261] The target network is obtained by neural network search;

[0262] The processing module is further configured to:

[0263] Obtain a plurality of candidate neural networks;

[0264] Select the target network from the plurality of candidate neural networks according to the performance of each candidate neural network; the performance is represented by the mean of the performance distribution and the standard deviation of the performance distribution.

[0265] Next, an execution device provided in an embodiment of the present application will be introduced. Please refer to Figure 20 , Figure 20 FIG. is a schematic structural diagram of an execution device provided in an embodiment of the present application. The execution device 2000 may specifically be embodied as a virtual reality (VR) device, a mobile phone, a tablet computer, a laptop computer, a smart wearable device, a monitoring data processing device, or a server, etc., which is not limited herein. Specifically, the execution device 2000 includes: a receiver 2001, a transmitter 2002, a processor 2003, and a memory 2004 (where the number of processors 2003 in the execution device 2000 may be one or more, Figure 20Take a processor as an example. Among them, the processor 2003 may include an application processor 20031 and a communication processor 20032. In some embodiments of the present application, the receiver 2001, the transmitter 2002, the processor 2003, and the memory 2004 may be connected through a bus or other means.

[0266] The memory 2004 may include a read-only memory and a random access memory, and provide instructions and data to the processor 2003. A part of the memory 2004 may also include a non-volatile random access memory (NVRAM). The memory 2004 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, where the operation instructions may include various operation instructions for implementing various operations.

[0267] The processor 2003 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, where the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all kinds of buses are referred to as a bus system in the figure.

[0268] The method disclosed in the embodiments of the present application can be applied to or implemented by the processor 2003. The processor 2003 can be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method can be completed through the integrated logic circuit of the hardware in the processor 2003 or instructions in software form. The above-mentioned processor 2003 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 2003 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 2004, and the processor 2003 reads the information in the memory 2004 and combines its hardware to complete the steps of the above method.

[0269] The receiver 2001 can be used to receive input digital or character information, and generate signal inputs related to the relevant settings and function controls of the execution device. The transmitter 2002 can be used to output digital or character information; the transmitter 2002 can also be used to send instructions to the disk group to modify the data in the disk group.

[0270] In the embodiments of the present application, in one case, the processor 2003 is used to execute the data processing method executed by the execution device in the above embodiments (such as the step of performing model inference through the target neural network).

[0271] The embodiments of the present application also provide a training device. Please refer to Figure 21 , Figure 21 is a schematic structural diagram of a training device provided by the embodiments of the present application. The training device 2100 can be deployed with Figure 19For the device described in the corresponding embodiment, specifically, the training device 2100 is implemented by one or more servers. The training device 2100 may vary greatly due to configuration or performance differences, and may include one or more central processing units (CPUs) 2121 (for example, one or more processors) and a memory 2132, and one or more storage media 2130 (for example, one or more mass storage devices) for storing application programs 2142 or data 2144. Among them, the memory 2132 and the storage media 2130 may be transient storage or persistent storage. The program stored in the storage media 2130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the training device. Further, the central processing unit 2121 may be configured to communicate with the storage media 2130 and execute a series of instruction operations in the storage media 2130 on the training device 2100.

[0272] The training device 2100 may further include one or more power supplies 2126, one or more wired or wireless network interfaces 2150, and one or more input / output interfaces 2158; or, one or more operating systems 2141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and so on.

[0273] In the embodiment of the present application, the central processing unit 2121 is used to execute Figure 12 the method in the corresponding embodiment.

[0274] In the embodiment of the present application, a computer program product is further provided. When it runs on a computer, it causes the computer to execute the steps performed by the foregoing data processing device, or causes the computer to execute the steps performed by the foregoing data processing device.

[0275] In the embodiment of the present application, a computer-readable storage medium is further provided. The computer-readable storage medium stores a program for signal processing. When it runs on a computer, it causes the computer to execute the steps performed by the foregoing data processing device, or causes the computer to execute the steps performed by the foregoing data processing device.

[0276] The execution device, training device, or terminal device provided in the embodiments of the present application may specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit may be a processor, for example, and the communication unit may be an input / output interface, a pin, a circuit, or the like. The processing unit may execute the computer execution instructions stored in the storage unit to cause the chip in the execution device to execute the data processing method described in the above embodiments, or to cause the chip in the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit outside the chip within the radio access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0277] Specifically, please refer to Figure 22 , Figure 22 which is a schematic structural diagram of the chip provided in the embodiments of the present application. The chip may be embodied as a neural network processor NPU 2200. The NPU 2200 is mounted as a coprocessor on the main CPU (Host CPU), and tasks are assigned by the Host CPU. The core part of the NPU is the arithmetic circuit 2203. The arithmetic circuit 2203 is controlled by the controller 2204 to extract matrix data from the memory and perform multiplication operations.

[0278] In some implementations, the arithmetic circuit 2203 includes multiple processing units (Process Engine, PE) inside. In some implementations, the arithmetic circuit 2203 is a two-dimensional systolic array. The arithmetic circuit 2203 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2203 is a general matrix processor.

[0279] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the corresponding data of matrix B from the weight memory 2202 and caches it on each PE in the arithmetic circuit. The arithmetic circuit fetches the data of matrix A from the input memory 2201 and performs matrix operations with matrix B, and the partial results or final results of the obtained matrix are stored in the accumulator 2208.

[0280] The unified memory 2206 is used to store input data and output data. The weight data is directly accessed by the Direct Memory Access Controller (DMAC) 2205 and is transferred to the weight memory 2202 by the DMAC. The input data is also transferred to the unified memory 2206 by the DMAC.

[0281] The BIU is the Bus Interface Unit, i.e., the bus interface unit 2210, which is used for the interaction between the AXI bus, the DMAC, and the Instruction Fetch Buffer (IFB) 2209.

[0282] The bus interface unit 2210 (Bus Interface Unit, abbreviated as BIU) is used for the instruction fetch buffer 2209 to obtain instructions from the external memory, and is also used for the storage unit access controller 2205 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0283] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 2206, or transfer the weight data to the weight memory 2202, or transfer the input data to the input memory 2201.

[0284] The vector calculation unit 2207 includes multiple arithmetic processing units, which, if necessary, further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for neural network non-convolution / full connection layer network calculations, such as Batch Normalization, pixel-level summation, upsampling of the feature plane, etc.

[0285] In some implementations, the vector calculation unit 2207 can store the processed output vector in the unified memory 2206. For example, the vector calculation unit 2207 can apply a linear function; or, a non-linear function to the output of the arithmetic circuit 2203, such as linear interpolation of the feature plane extracted by the convolutional layer, or a vector of accumulated values, to generate activation values. In some implementations, the vector calculation unit 2207 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as the activation input to the arithmetic circuit 2203, such as for use in subsequent layers in the neural network.

[0286] The instruction fetch buffer 2209 connected to the controller 2204 is used to store the instructions used by the controller 2204;

[0287] The unified memory 2206, the input memory 2201, the weight memory 2202, and the fetch memory 2209 are all On-Chip memories. The external memory is private to the NPU hardware architecture.

[0288] Wherein, the processor mentioned anywhere above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0289] In addition, it should be noted that the device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines.

[0290] Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by means of dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits, or dedicated circuits. However, for this application, in more cases, software program implementation is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a floppy disk, a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disc of a computer, and includes several instructions for causing a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in various embodiments of this application.

[0291] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0292] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center by wire (such as coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

Claims

1. A data processing method, characterized in that, The method includes: Based on the input data of the convolutional layer, a mask of the input data is obtained through a prediction network; the input data includes data in the channel dimension and the spatial dimension; the mask includes a mask in the channel dimension and a mask in the spatial dimension; Based on the mask in the channel dimension, a prefix sum in the channel dimension is obtained; Based on the mask in the spatial dimension, a prefix sum in the spatial dimension when performing a single sliding window convolution on the input data by the convolution kernel of the convolutional layer is obtained; Based on the fusion result of the prefix sum in the channel dimension and the prefix sum in the spatial dimension, the mask is applied to the input data, and the masked input data is processed by the convolution kernel of the convolutional layer for the sliding window convolution.

2. The method according to claim 1, characterized in that, The fusion result is the Cartesian product of the prefix sum in the channel dimension and the prefix sum in the spatial dimension.

3. The method according to claim 1 or 2, characterized in that The method further includes: Based on the mask in the spatial dimension, a prefix sum in the spatial dimension when performing another sliding window convolution on the input data by the convolution kernel of the convolutional layer is obtained; Based on the fusion result of the prefix sum in the channel dimension and the prefix sum in the spatial dimension during the another sliding window convolution, the mask is applied to the input data, and the masked input data is processed by the convolution kernel of the convolutional layer for the another sliding window convolution.

4. The method according to any one of claims 1 to 3, characterized in that, The obtaining of the prefix sum in the channel dimension based on the mask in the channel dimension includes: Based on the mask in the channel dimension, a prefix sum in the channel dimension is obtained through a first hardware unit; The obtaining of the prefix sum in the spatial dimension when performing a single sliding window convolution on the input data by the convolution kernel of the convolutional layer based on the mask in the spatial dimension includes: Based on the mask in the spatial dimension, a prefix sum in the channel dimension is obtained through a second hardware unit, and the first hardware unit and the second hardware unit are different.

5. The method according to any one of claims 1 to 4, characterized in that The method further includes: Through a third hardware unit, the prefix sum in the channel dimension and the prefix sum in the spatial dimension are fused to obtain the fusion result, and the third hardware unit is different from the first hardware unit and the second hardware unit.

6. The method according to any one of claims 1 to 5, characterized in that The method further includes: The image data is processed through a first network to obtain the input data of the convolutional layer; The output data of the convolutional layer is processed through a second network to obtain the processing result of the image data. The first network, the second network, and the convolutional layer belong to a target network, and the target network is used to process the image data for a target task to obtain the processing result. The target task is one of image classification, object recognition, text recognition in an image, image segmentation, and image enhancement.

7. The method according to any one of claims 1 to 6, characterized in that, The method is applied to the data processing process of the target network; The target network is obtained through neural network search; The method further includes: Obtaining a plurality of candidate neural networks; Based on the performance of each candidate neural network, the target network is selected from the plurality of candidate neural networks; the performance is represented by the mean of the performance distribution and the standard deviation of the performance distribution.

8. The method according to claim 7, wherein The mean value and the standard deviation are determined by Bayesian inference and Monte Carlo sampling.

9. A data processing device, characterized in that, The device includes: a processing module, configured to obtain a mask of the input data through a prediction network according to the input data of the convolutional layer; the input data includes data in a channel dimension and a spatial dimension; the mask includes a mask in the channel dimension and a mask in the spatial dimension; a first hardware unit, configured to obtain a prefix sum in the channel dimension according to the mask in the channel dimension; a second hardware unit, configured to obtain a prefix sum in the spatial dimension when performing a single sliding window convolution on the input data by a convolution kernel of the convolutional layer according to the mask in the spatial dimension; the processing module is further configured to, according to a fusion result of the prefix sum in the channel dimension and the prefix sum in the spatial dimension, apply the mask to the input data, and perform processing of the sliding window convolution on the masked input data by the convolution kernel of the convolutional layer.

10. The device according to claim 9, characterized in that, The fusion result is a Cartesian product of the prefix sum in the channel dimension and the prefix sum in the spatial dimension.

11. The device according to claim 9 or 10, characterized in that The second hardware unit is further configured to obtain a prefix sum in the spatial dimension when performing another sliding window convolution on the input data by the convolution kernel of the convolutional layer according to the mask in the spatial dimension; the processing module is further configured to, according to a fusion result of the prefix sum in the channel dimension and the prefix sum in the spatial dimension during the another sliding window convolution, apply the mask to the input data, and perform processing of the another sliding window convolution on the masked input data by the convolution kernel of the convolutional layer.

12. The device according to any one of claims 9 to 11, characterized in that, The processing module is further configured to: process the image data through a first network to obtain the input data of the convolutional layer; process the output data of the convolutional layer through a second network to obtain a processing result of the image data, where the first network, the second network, and the convolutional layer belong to a target network, and the target network is configured to process the image data for a target task to obtain the processing result, and the target task is one of image classification, target recognition, text recognition in an image, image segmentation, and image enhancement.

13. The device according to any one of claims 9 to 12, characterized in that, The device is applied to a data processing process of a target network; The target network is obtained by neural network search; The processing module is further configured to: acquire a plurality of candidate neural networks; select the target network from the plurality of candidate neural networks according to the performance of each candidate neural network; the performance is represented by a mean value of a performance distribution and a standard deviation of the performance distribution.

14. The device according to any one of claims 9 to 13, characterized in that, The mean value and the standard deviation are determined by Bayesian inference and Monte Carlo sampling.

15. A computing device, characterized in that, including at least one processor and at least one memory, where computer-readable instructions are stored in the at least one memory; the at least one processor executes the computer-readable instructions so that the computing device executes the method according to any one of claims 1 to 8.

16. A computer-readable storage medium, characterized in that, including computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 8.

17. A computer program product, characterized in that, Comprising computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 8.

18. A chip, characterized in that, Comprising at least one processing unit and an interface circuit, the interface circuit is used to provide program instructions or data for the at least one processing unit, the at least one processing unit is used to execute the program instructions to implement the method according to any one of claims 1 to 8, the at least one processing unit includes a first hardware unit and a second hardware unit, the first hardware unit is used to calculate the prefix sum in the channel dimension, and the second hardware unit is used to calculate the prefix sum in the spatial dimension.

Images