Data processing method and apparatus therefor

Abstract

A data processing method, applied to the field of artificial intelligence, the method comprising: on the basis of a first value and by means of a first computing unit, obtaining a plurality of sub-values, the different sub-values being digits in the first value at different order-of-magnitude positions, the data type of the first value being different from the data type of a model parameter of a network layer, the data type of each sub-value being the same as the data type of the model parameter, and the model parameter being compressed data; by means of a second computing unit, performing computation on the model parameter and each sub-value, so as to obtain a plurality of operation results; and, by means of the first computing unit, combining the plurality of operation results, so as to obtain a second value, the data type of the second value being the same as the data type of the first value. The present application can reduce the amount of data transfer, thereby improving the operation efficiency.

Description

Data processing method and device thereof

[0001] The present application claims priority to the Chinese Patent Application No. 202410996082.3, filed on July 23, 2024, and entitled "A data processing method and device thereof", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a data processing method and device thereof. BACKGROUND

[0003] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, enabling machines to have perception, reasoning and decision-making functions.

[0004] Large model inference accelerates the advancement of neural networks in academia and industry. Large model inference can extract valuable information from complex models and solve a variety of problems such as image recognition, speech recognition, natural language processing, etc. However, the inference of large models usually requires a large amount of computing resources, which to some extent limits the universality and real-time performance of its application. Therefore, how to accelerate the inference of large models and improve their efficiency in practical applications is a problem that needs to be solved. SUMMARY

[0005] In a first aspect, the present application provides a data processing method, comprising: obtaining a plurality of sub-values by a first calculation unit according to a first value, wherein different sub-values are digits located at different orders of magnitude positions in the first value, the data type of the first value is different from the data type of a model parameter of a network layer, the data type of each sub-value is the same as the data type of the model parameter, and the model parameter is compressed data; calculating the model parameter and each sub-value by a second calculation unit to obtain a plurality of operation results; and merging the plurality of operation results by the first calculation unit to obtain a second value, wherein the data type of the second value is the same as the data type of the first value.

[0006] In the embodiments of the present application, in order to solve the problem of large memory reading caused by the large amount of data in the memory after the inverse quantization operation, the baseline scheme of "decompressing before compressing data" is changed to "directly processing in the compressed space". Specifically, the input data (not the model parameters) is transformed to obtain multiple sub-values consistent with the data type of the model parameters, and the model parameters and each sub-value are operated, and then the operation result is merged to obtain the output. Since no inverse quantization processing is involved, that is, the values are all in the compressed space, therefore, the data amount of the first data unit and the second data unit when obtaining data is small (not the data after inverse quantization), thereby reducing the data carrying amount and improving the operation efficiency.

[0007] In a possible implementation, the data type of the first value is a floating point type, and the sub-value is an integer type. In addition, it is not necessarily limited to splitting the floating point number into an integer, for example, the floating point number can also be split into a more fine-grained floating point number, or the integer can be split into a more fine-grained integer.

[0008] In a possible implementation, the number of bits of the data type of the first value is greater than the number of bits of the data type of each sub-value.

[0009] In a possible implementation, the first value is a floating point type, and the plurality of sub-values includes a first sub-value and a second sub-value; wherein the first sub-value is a part of the digits of the decimal of the first value, and the second sub-value is a part of the digits of the decimal of the first value. That is, the decimal of the first value can be split into multiple sub-values to represent (that is, the decimal is split in multiple scales).

[0010] In addition, the multi-scale splitting here is not necessarily according to the "integer part, decimal part" under the decimal system, and can be adapted to other splitting means under other number systems.

[0011] In a possible implementation, the first value is a floating point type, and the plurality of sub-values includes a first sub-value and a second sub-value; wherein the first sub-value is all the digits of the decimal of the first value, or the second sub-value is all the digits of the integer of the first value. That is, the integer or the whole of the decimal of the first value can be represented by a sub-value.

[0012] Of course, the integer can be split in multiple scales, and the whole of the decimal can be represented by a sub-value, and the decimal can be split in multiple scales, and the whole of the integer can be represented by a sub-value.

[0013] In a possible implementation, the method further includes:

[0014] carrying the first value from the memory to the first calculation unit;

[0015] carrying the plurality of sub-values obtained by the first calculation unit to the memory;

[0016] carrying the plurality of sub-values and the model parameter from the memory to the second calculation unit;

[0017] carrying the plurality of calculation results obtained by the second calculation unit to the memory;

[0018] carrying the plurality of calculation results from the memory to the first calculation unit.

[0019] In a possible implementation, the memory is a global memory (GM), the first calculation unit is a vector calculation unit, and the second calculation unit is a matrix calculation unit.

[0020] In a possible implementation, the calculating, by the second calculation unit, the model parameter and each of the sub-values respectively includes performing a multiplication operation on the model parameter and each of the sub-values by the second calculation unit.

[0021] In a second aspect, a data processing apparatus is provided, and the apparatus includes:

[0022] a first calculation unit configured to obtain a plurality of sub-values according to a first value, wherein different sub-values are digits located at different orders of magnitude in the first value, the data type of the first value is different from that of a model parameter of a network layer, the data type of each of the sub-values is the same as that of the model parameter, and the model parameter is compressed data;

[0023] a second calculation unit configured to calculate the model parameter and each of the sub-values respectively to obtain a plurality of calculation results;

[0024] The first calculation unit is further configured to combine the plurality of calculation results to obtain a second value, and the data type of the second value is the same as that of the first value.

[0025] In a possible implementation, the data type of the first value is a floating point type, and the sub-values are integer types.

[0026] In a possible implementation, the number of bits of the data type of the first value is greater than that of the data type of each of the sub-values.

[0027] In a possible implementation, the first value is a floating point type, and the plurality of sub-values include a first sub-value and a second sub-value; wherein,

[0028] the first sub-value is a part of the integer of the first value, and the second sub-value is a part of the integer of the first value; or

[0029] the first sub-value is a part of the integer of the first value, and the second sub-value is a part of the integer of the first value; or

[0030] the first sub-value is a part of the integer of the first value, and the second sub-value is a part of the integer of the first value; or

[0031] In a possible implementation, the apparatus further includes:

[0032] a transceiving module, configured to transfer the first value from the memory to the first calculation unit;

[0033] a transferring module, configured to transfer the plurality of sub-values obtained by the first calculation unit to the memory;

[0034] a transferring module, configured to transfer the plurality of sub-values and the model parameter from the memory to the second calculation unit;

[0035] a transferring module, configured to transfer the plurality of calculation results obtained by the second calculation unit to the memory;

[0036] a transferring module, configured to transfer the plurality of calculation results from the memory to the first calculation unit.

[0037] In a possible implementation, the memory is a global memory (GM), the first calculation unit is a vector calculation unit, and the second calculation unit is a matrix calculation unit.

[0038] In a possible implementation, the second calculation unit is further configured to perform a product operation on the model parameter and each of the sub-values.

[0039] In a third aspect, an embodiment of the present application provides a data processing apparatus, which can include a memory, a processor and a bus system, wherein the memory is configured to store a program, and the processor is configured to execute the program in the memory to perform the method in the first aspect and any optional method thereof.

[0040] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is run on a computer, the computer program causes the computer to perform the method in the first aspect and any optional method thereof.

[0041] In a fifth aspect, an embodiment of the present application provides a computer program, which, when run on a computer, causes the computer to perform the method in the first aspect and any optional method thereof.

[0042] Sixthly, this application provides a chip system including a processor for supporting an execution data processing device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0043] Figure 1A is a schematic diagram of a structural framework for artificial intelligence.

[0044] Figures 1B and 1C are schematic diagrams of the application system framework of the present invention;

[0045] Figure 1D is a schematic diagram of an optional hardware structure for the terminal;

[0046] Figure 2 is a schematic diagram of a server structure;

[0047] Figure 3 is a schematic diagram of a system architecture according to this application;

[0048] Figure 4 illustrates the process of a cloud service.

[0049] Figure 5 is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0050] Figures 6A and 6B are schematic diagrams of a system architecture according to this application;

[0051] Figure 7 is a schematic diagram of a data processing method provided in an embodiment of this application;

[0052] Figure 8 is a schematic diagram of a data processing method provided in an embodiment of this application;

[0053] Figure 9 is a schematic diagram of a framework provided in an embodiment of this application;

[0054] Figure 10 is a schematic diagram of a data processing device provided in an embodiment of this application;

[0055] Figure 11 is a schematic diagram of an execution device provided in an embodiment of this application;

[0056] Figure 12 is a schematic diagram of a training device provided in an embodiment of this application;

[0057] Figure 13 is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0058] The embodiments of the present application will be described below in conjunction with the accompanying drawings. The terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0059] The embodiments of the present application will be described below in conjunction with the accompanying drawings. The terms used in the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0060] The terms "first", "second", and the like in the description of the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged, as appropriate, and that this is merely a way of distinguishing between objects of the same attribute used in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus including a series of units does not necessarily limit to those units, but can include other units not clearly listed or inherent to the process, method, product, or apparatus.

[0061] The terms "substantially", "about", and the like used herein are used as approximate terms, not as degree terms, and are intended to take into account the inherent deviations of measured or calculated values known to those skilled in the art. In addition, the use of "may" in describing the embodiments of the present application means "one or more embodiments". The terms "use", "using", and "used" used herein can be considered synonymous with the terms "utilize", "utilizing", and "utilized", respectively. In addition, the term "exemplary" is intended to refer to an example or an illustration.

[0062] First, the overall workflow of the artificial intelligence system is described, please refer to FIG. 1A, which shows a structural diagram of an artificial intelligence subject framework, and the following describes the artificial intelligence subject framework from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0063] (1) Infrastructure

[0064] The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network related platform guarantee and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication acquire data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for calculation.

[0065] (2) Data

[0066] The data on the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. Data involves graphics, images, speech, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, humidity, etc.

[0067] (3) Data processing

[0068] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0069] Among them, machine learning and deep learning can symbolize and formalize intelligent information modeling, extraction, preprocessing, training, etc.

[0070] Reasoning refers to the process of simulating human intelligent reasoning methods in computers or intelligent systems, using formalized information to perform machine thinking and solve problems according to reasoning control strategies, and the typical function is search and matching.

[0071] Decision-making refers to the process of decision-making after intelligent information reasoning, which usually provides functions such as classification, sorting, prediction, etc.

[0072] (4) General capabilities

[0073] After the data is processed as mentioned above, further based on the results of data processing, some general capabilities can be formed, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0074] (5) Intelligent products and industry applications

[0075] Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields, which are the packaging of overall solutions of artificial intelligence, and realize the application of intelligent information decision products. Its application fields mainly include: intelligent terminal, intelligent transportation, intelligent medical treatment, automatic driving, smart city, etc.

[0076] The present application can be applied to the field of natural language processing in the field of artificial intelligence. The following will introduce multiple application scenarios landed in products by taking natural language processing as an example.

[0077] First, the application scenarios of the present application are introduced. The present application can be but is not limited to the application of generative artificial intelligence (artificial intelligence generated content, AIGC) function (hereinafter referred to as synthetic application) or cloud service provided by cloud-side server, etc., which will be introduced as follows:

[0078] I. Synthetic application

[0079] The product form of the embodiments of the present application can be a synthetic application. The synthetic application can run on a terminal device or a server on the cloud side.

[0080] In one possible implementation, the synthetic application can implement a data generation task based on input data (such as images, texts, audios, videos, etc.). The synthetic application can execute the data generation task in response to the input input data (such as images, texts, audios, videos, etc.) to obtain generated data.

[0081] For example, the above-mentioned data generation task can be but is not limited to:

[0082] Text generation task: It can generate various types of text content, including news reports, blog articles, product descriptions, social media posts, etc. It can generate logically coherent and coherent text according to given topics and requirements.

[0083] Image generation task: Can generate images, including illustrations, artwork, design drafts, etc. It can generate image content related to the given description or keywords.

[0084] Audio generation task: Can generate speech content, including reading text, voice assistant answers, etc. It can simulate human voice characteristics and intonation, making the generated speech sound more natural.

[0085] Content summary and summary task: Can read a large amount of text content and generate summaries or summaries. It can extract key information from the text and present it to the user in a concise manner.

[0086] Language translation task: Can perform language translation, translating text in one language into another language. It can handle multiple language pairs and provide accurate translation results.

[0087] Automatic reply and customer service: Can be used to automatically reply to user questions and provide customer service. It can understand the user's intent and give accurate answers or suggestions.

[0088] In one possible implementation, the user can open the synthetic application installed on the terminal device and input the input data (such as images, text, audio, video, etc.), and the synthetic application can generate data for the input data through the method provided in the present application. Embodiments, and the generated data is presented to the user (the presentation method can be but is not limited to display, save, upload to the cloud side, etc.).

[0089] In one possible implementation, the user can open the synthetic application installed on the terminal device and input the input data, and the synthetic application can send the input data to the server on the cloud side, and the server on the cloud side can generate data for the input data through the method provided in the present application. Embodiments, and the generated data is transmitted back to the terminal device, and the terminal device can present the generated data to the user (the presentation method can be but is not limited to display, save, upload to the cloud side, etc.).

[0090] Next, the synthetic application in the present application is introduced from the functional architecture and the product architecture that realizes the function.

[0091] Referring to FIG. 1B, FIG. 1B is a functional architecture of the synthetic application in the present application:

[0092] In a possible implementation, as shown in FIG. 1B, the synthetic application 102 can receive an input parameter 101 (for example, containing input data) and generate generated data 103. The synthetic application 102 can be executed on, for example, at least one computer system, and includes computer code that, when executed by one or more computers, causes the computers to perform the natural language model trained by the method provided by the embodiments of the present application.

[0093] Referring to FIG. 1C, FIG. 1C is a schematic diagram of an entity architecture running a synthetic application in embodiments of the present application:

[0094] Referring to FIG. 1C, FIG. 1C is a schematic diagram of a system architecture. The system can include a terminal 100 and a server 200. The server 200 can include one or more servers (one server is shown in FIG. 1C as an example), and the server 200 can provide a synthesis function service for one or more terminals.

[0095] The terminal 100 can install a synthetic application or open a webpage related to the synthesis function. The application and the webpage can provide an interface, and the terminal 100 can receive parameters input by a user on the synthesis function interface and send the parameters to the server 200. The server 200 can obtain a processing result based on the received parameters and return the processing result to the terminal 100.

[0096] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining a processing result based on received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited thereto.

[0097] Next, the product form of the terminal 100 in FIG. 1C is described.

[0098] The terminal 100 in the embodiments of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like, and the embodiments of the present application are not limited thereto.

[0099] FIG. 1D shows a schematic diagram of an optional hardware structure of the terminal 100.

[0100] Referring to FIG. 1D, the terminal 100 can include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a processor 170, an external interface 180, a power supply 190, and the like. It will be understood by those skilled in the art that FIG. 1D is merely an example of a terminal or a multi-functional device and does not limit the terminal or the multi-functional device, which can include more or less components than those shown, or can have different arrangements of the components, or different names of the components.

[0101] The input unit 130 can be used to receive inputted digital or character information, and to generate key signal input with respect to user setting and function control of the portable multi-functional device. Specifically, the input unit 130 can include a touch screen 131 (optional) and / or other input device 132. The touch screen 131 can collect a touch operation of a user thereon or therearound (such as an operation of a user using a finger, a knuckle, a stylus, or any suitable object on or near the touch screen), and drive a corresponding connected device according to a pre-set program. The touch screen can detect a touch action of a user on the touch screen, convert the touch action into a touch signal and send the touch signal to the processor 170, and can receive a command from the processor 170 and execute the command; the touch signal at least includes touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can include other input devices. Specifically, the other input device 132 can include, but is not limited to, one or more of a physical keyboard, a function key (such as a volume control button 132, an on / off button 133, and the like), a trackball, a mouse, a joystick, and the like.

[0102] The input device 132 can receive input data and the like.

[0103] The display unit 140 can be used to display information input by a user or information provided to a user, various menus of the terminal 100, an interactive interface, file display, and / or playing of any kind of multimedia file. In the embodiments of the present application, the display unit 140 can be used to display an interface of a synthetic application program, generated generation data, and the like.

[0104] The memory 120 can be used to store instructions and data. The memory 120 can mainly include a storage instruction area and a storage data area. The storage data area can store various data such as multimedia files, texts, etc. The storage instruction area can store software units such as operating systems, applications, instructions required by at least one function, etc. or their subsets, extended sets. Non-volatile random access memory can also be included. The processor 170 is provided with software and applications that include management of hardware, software and data resources in the computing processing device, support control. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0105] The processor 170 is the control center of the terminal 100, which connects each part of the entire terminal 100 through various interfaces and lines, executes various functions of the terminal 100 and processes data by running or executing instructions stored in the memory 120 and calling data stored in the memory 120, thereby performing overall control of the terminal device. Optionally, the processor 170 can include one or more processing units. Preferably, the processor 170 can integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 170. In some embodiments, the processor, the memory, can be implemented on a single chip, and in some embodiments, they can also be implemented on separate chips respectively. The processor 170 can also be used to generate corresponding operation control signals to the corresponding components of the computing processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0106] The memory 120 can be used to store instructions and data. The memory 120 can mainly include a storage instruction area and a storage data area. The storage data area can store various data such as multimedia files, texts, etc. The storage instruction area can store software units such as operating systems, applications, instructions required by at least one function, etc. or their subsets, extended sets. Non-volatile random access memory can also be included. The processor 170 is provided with software and applications that include management of hardware, software and data resources in the computing processing device, support control. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0107] The RF unit 110 (optional) can be used to receive and send signals in information or communication processes. For example, after receiving the downlink information from the base station, the processor 170 processes the information. In addition, the uplink data is sent to the base station. Generally, the RF circuit includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0108] In the embodiments of the present application, the RF unit 110 can send input data to the server 200 and receive generated data sent by the server 200.

[0109] It should be understood that the RF unit 110 is optional, which can be replaced by other communication interfaces, for example, a network interface.

[0110] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can realize the functions of managing charging, discharging, and power consumption management, etc.

[0111] The terminal 100 also includes an external interface 180, which can be a standard Micro USB interface, or a multi-pin connector. The external interface 180 can be used to connect the terminal 100 to other devices for communication, or to connect a charger to charge the terminal 100.

[0112] Although not shown, the terminal 100 can also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, different function sensors, etc., which will not be described here. Some or all of the methods described below can be applied in the terminal 100 as shown in FIG. 1D.

[0113] Next, the product form of the server 200 in FIG. 1C is described;

[0114] FIG. 2 provides a structural schematic diagram of a server 200. As shown in FIG. 2, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate through the bus 201.

[0115] The bus 201 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is shown in FIG. 2, but it does not mean that there is only one bus or only one type of bus.

[0116] The processor 202 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0117] The memory 204 can include a volatile memory, such as a random access memory (RAM). The memory 204 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard drive (HDD), or a solid state drive (SSD).

[0118] The memory 204 can be used to store software codes related to the data processing method, and the processor 202 can execute the steps of the chip data processing method or schedule other units to realize the corresponding functions.

[0119] It should be understood that the terminal 100 and the server 200 described above can be centralized or distributed devices, and the processors (for example, the processor 170 and the processor 202) in the terminal 100 and the server 200 can be hardware circuits (for example, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor, a microcontroller, or the like) or a combination of the hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, a DSP, or the like, or a hardware system without an instruction execution function, such as an ASIC, an FPGA, or the like, or a combination of the hardware system without an instruction execution function and the hardware system with an instruction execution function.

[0120] It should be understood that the steps related to the model inference process in the embodiments of the present application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and the server is not limited to the architecture of the processor combined with the memory described above. The system architecture provided by the embodiments of the present application will be described in detail below with reference to FIG. 5.

[0121] FIG. 5 is a schematic diagram of a system architecture provided by an embodiment of the present application. As shown in FIG. 5, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data collection system 560.

[0122] The execution device 510 includes a computing module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The target model / rule 501 can be included in the computing module 511, and the preprocessing module 513 and the preprocessing module 514 are optional.

[0123] The execution device 510 can be a terminal device or a server that runs a synthetic application program.

[0124] The data collection device 560 is configured to collect training samples. The training samples can be program files (including program codes and program input data) and the like. After the training samples are collected, the data collection device 560 stores the training samples in the database 530.

[0125] The training device 520 can train a neural network to be trained based on the training samples maintained in the database 530 to obtain the target model / rule 501.

[0126] It should be noted that, in actual application, the training samples maintained in the database 530 do not necessarily all come from the collection of the data collection device 560, but can also be received from other devices. In addition, it should be noted that the training device 520 does not necessarily train the target model / rule 501 based on the training samples maintained in the database 530, but can also obtain training samples from the cloud or other places for model training. The above description should not be regarded as a limitation of the embodiments of the present application.

[0127] The target model / rule 501 trained by the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in FIG. 5. The execution device 510 can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle-mounted terminal, etc., and can also be a server, etc.

[0128] Specifically, the training device 520 can deliver the trained model to the execution device 510.

[0129] In FIG. 5, the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with external devices. The user can input data (for example, input data in the embodiments of the present application) to the I / O interface 512 through the client device 540.

[0130] The pre-processing modules 513 and 514 are used for pre-processing the input data received by the I / O interface 512. It should be understood that there can be no pre-processing modules 513 and 514 or only one pre-processing module. When there is no pre-processing module 513 and 514, the input data can be directly processed by the calculation module 511.

[0131] During the pre-processing of the input data by the execution device 510 or during the calculation process performed by the calculation module 511 of the execution device 510, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained by the corresponding processing in the data storage system 550.

[0132] Finally, the I / O interface 512 provides the processing result (for example, generated data, etc.) to the client device 540, thereby providing the user.

[0133] In the case shown in FIG. 5, the user can manually give input data, which can be operated through the interface provided by the I / O interface 512. In another case, the client device 540 can automatically send input data to the I / O interface 512, and if the user's authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permission in the client device 540. The user can view the results output by the execution device 510 in the client device 540, and the specific presentation form can be a specific way such as display, sound, action, etc. The client device 540 can also act as a data collection end, collect the input data input into the I / O interface 512 and the output results output from the I / O interface 512 as new sample data, and store them into the database 530. Of course, the collection can also be performed without the client device 540, and the I / O interface 512 can directly store the input data input into the I / O interface 512 and the output results output from the I / O interface 512 as new sample data into the database 530.

[0134] It is worth noting that FIG. 5 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules and the like shown in the figure does not constitute any limitation. For example, in FIG. 5, the data storage system 550 is an external memory relative to the execution device 510, and in other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the above execution device 510 can be deployed in the client device 540.

[0135] From the inference side of the model:

[0136] In the embodiment of the present application, the computing module 511 of the execution device 510 can obtain the code stored in the data storage system 550 to implement the steps related to the model inference process in the embodiment of the present application.

[0137] In the embodiment of the present application, the computing module 511 of the execution device 510 can include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 can be a hardware system with an execution instruction function, such as a CPU, a DSP, etc., or a hardware system without an execution instruction function, such as an ASIC, an FPGA, etc., or a combination of the hardware system without an execution instruction function and the hardware system with an execution instruction function.

[0138] Specifically, the computing module 511 of the execution device 510 can be a hardware system with an execution instruction function, and the steps related to the model inference process provided in the embodiments of the present application can be software codes stored in the memory. The computing module 511 of the execution device 510 can obtain the software codes from the memory and execute the obtained software codes to implement the steps related to the model inference process provided in the embodiments of the present application.

[0139] It should be understood that the computing module 511 of the execution device 510 can be a combination of a hardware system without an execution instruction function and a hardware system with an execution instruction function, and part of the steps related to the model inference process provided in the embodiments of the present application can also be implemented by the hardware system without an execution instruction function in the computing module 511 of the execution device 510, which is not limited here.

[0140] From the training side of the model:

[0141] In the embodiments of the present application, the training device 520 can obtain the codes stored in the memory (not shown in FIG. 5, which can be integrated in the training device 520 or deployed separately from the training device 520) to implement the steps related to the model training in the embodiments of the present application.

[0142] In the embodiments of the present application, the training device 520 can include hardware circuits (such as application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.), or combinations of these hardware circuits. For example, the training device 520 can be a hardware system with an execution instruction function, such as a CPU, a DSP, etc., or a hardware system without an execution instruction function, such as an ASIC, an FPGA, etc., or a combination of the hardware system without an execution instruction function and the hardware system with an execution instruction function.

[0143] It should be understood that the training device 520 can be a combination of a hardware system without an execution instruction function and a hardware system with an execution instruction function, and part of the steps related to the model training provided in the embodiments of the present application can also be implemented by the hardware system without an execution instruction function in the training device 520, which is not limited here.

[0144] II. Server-provided integration function cloud service:

[0145] In a possible implementation, the server can provide a service of the integration function for the end side through an application programming interface (API).

[0146] The terminal device can send relevant parameters (for example, containing input data) to the server through the API provided by the cloud, and the server can obtain a processing result (for example, generated data) based on the received parameters and return the processing result to the terminal.

[0147] The description of the terminal and the server can be described in the above embodiments, which will not be described here.

[0148] As shown in FIG. 6A, a process of using a cloud service of the integration function provided by a cloud platform is shown.

[0149] 1. Open and purchase the content review service.

[0150] 2. The user can download the software development kit (SDK) corresponding to the content review service. The cloud platform usually provides multiple development versions of the SDK for the user to select according to the needs of the development environment, such as the JAVA version of the SDK, the python version of the SDK, the PHP version of the SDK, the Android version of the SDK, etc.

[0151] 3. The user downloads the corresponding version of the SDK to the local according to the needs, imports the SDK project to the local development environment, and configures and debugs in the local development environment. The local development environment can also develop other functions, so as to form an application that integrates the integration function class capability.

[0152] 4. During the use of the integration function class application, when the integration function is needed, the API call of the integration function can be triggered. When the application triggers the integration function, an API request is initiated to the running instance of the integration function class service in the cloud environment, wherein the API request carries input data. The running instance in the cloud environment processes the input data to obtain a processing result (for example, generated data).

[0153] 5. The cloud environment returns the processing result to the application, thereby completing one integration function service call.

[0154] In addition to the application program and the cloud service, the implementation form of the present application can also be a large model inference acceleration library and a large model application SDK.

[0155] In order to better understand the scheme of the embodiments of the present application, the following takes text generation as an example to briefly introduce the possible application scenarios of the embodiments of the present application in combination with FIGS. 2 to 4.

[0156] Figure 3 shows a natural language processing system, which includes a user device and a data processing device. The user device includes a smart terminal such as a mobile phone, a personal computer, or an information processing center. The user device is the initiator of natural language data processing, and is the initiator of language query or inquiry. Usually, the user initiates the request through the user device.

[0157] The data processing device can be a cloud server, a network server, an application server, a management server, or other devices or servers with data processing functions. The data processing device receives query statements, voices, texts, etc. from the smart terminal through the interactive interface, and performs language data processing in the form of machine learning, deep learning, searching, reasoning, decision-making, etc. through the storage memory and the data processing processor, and feeds back the processing results to the user device. The storage memory in the data processing device can be a general term, including local storage and database storing historical data. The database can be on the data processing device or on other network servers.

[0158] In the natural language processing system shown in Figure 3, the user device can receive the user's instructions, for example, the user device can receive a piece of text input by the user, and then initiate a request to the data processing device, so that the data processing device performs natural language processing applications (such as natural language generation, text classification, text reasoning, named entity recognition, translation, etc.) on the piece of text obtained by the user device, thereby obtaining the processing results (such as predicted word results, classification results, reasoning results, named entity recognition results, translation results, etc.) of the corresponding natural language processing applications on the piece of text.

[0159] In the embodiments of the present application, the user device can receive the user's instructions, for example, the user device can receive a piece of text (such as input data) input by the user, and then initiate a request to the data processing device, so that the data processing device performs natural language processing applications (such as text synthesis, etc.) on the piece of text obtained by the user device, thereby obtaining the processing results (such as generated data, etc.) of the corresponding natural language processing applications on the piece of text.

[0160] In Figure 3, the data processing device can process the above-mentioned text data by the method provided in the embodiments of the present application.

[0161] Figure 4 shows another natural language processing system, in which the user device directly serves as the data processing device. The user device can directly receive input from the user and directly process the input by the hardware of the user device itself. The specific process is similar to that of Figure 3, and reference can be made to the above description, which will not be repeated here.

[0162] FIG. 4 is a schematic diagram of a device related to natural language processing according to an embodiment of the present application.

[0163] The processor in FIG. 3 and FIG. 4 can train data / machine learn / deep learn through a neural network model or other model, and use the model (e.g., a natural language model in an embodiment of the present application, etc.) finally trained or learned by data to perform natural language processing application (e.g., program synthesis, etc.) on text data (e.g., input data text described in an embodiment of the present application), so as to obtain corresponding processing results.

[0164] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the related terms and concepts related to neural networks involved in the embodiments of the present application will be introduced first.

[0165] (1) Neural network

[0166] The neural network can be composed of neural units, and the neural unit can be an operation unit with xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0167] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. The neural network is a network formed by connecting a plurality of the above single neural units, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of a plurality of neural units.

[0168] (2) Transformer layer

[0169] The neural network comprises an embedding layer and at least one transformer layer, the at least one transformer layer can be N transformer layers (N is an integer greater than 0), wherein each transformer layer comprises, in sequence, an attention layer, an add&norm layer, a feed forward layer and an add&norm layer. In the embedding layer, the current input is embedded to obtain a plurality of embedding vectors; in the attention layer, P input vectors are obtained from the previous layer of the first transformer layer, and a first input vector in the P input vectors is taken as a center, and based on the correlation between each input vector in a preset attention window range and the first input vector, an intermediate vector corresponding to the first input vector is obtained, and thus P intermediate vectors corresponding to the P input vectors are determined; in the pooling layer, the P intermediate vectors are combined into Q output vectors, and the plurality of output vectors obtained by the last transformer layer in the transformer layer are used as the feature representation of the current input.

[0170] (3) Attention mechanism

[0171] The attention mechanism simulates the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external feeling to increase the observation fineness of a part of the region, and can quickly filter out high-value information from a large amount of information using limited attention resources. The attention mechanism can quickly extract important features of sparse data, and is widely used in natural language processing tasks, especially machine translation. The self-attention mechanism is an improvement of the attention mechanism, which reduces the dependence on external information and is better at capturing the internal correlation of data or features. The essential idea of the attention mechanism can be rewritten as the following formula:

[0172] Wherein, Lx=||Source|| represents the length of Source, the formula means that the elements in Source are imagined to be composed of a series of data pairs, at this time, given an element Query in the target Target, by calculating the similarity or correlation between Query and each Key, the weight coefficient of the Value corresponding to each Key is obtained, and then the Value is weighted and summed, that is, the final Attention value is obtained. Therefore, the essence of Attention mechanism is to weight and sum the Value values of the elements in Source, and Query and Key are used to calculate the weight coefficient of the corresponding Value. Conceptually, Attention can be understood as selectively filtering out a small amount of important information from a large amount of information and focusing on these important information, ignoring a large amount of unimportant information. The focusing process is reflected in the calculation of the weight coefficient, the greater the weight, the more focused on the corresponding Value value, that is, the weight represents the importance of the information, and the Value is the corresponding information. The self-attention mechanism can be understood as intra attention. Attention mechanism occurs between the element Query in the Target and all elements in the Source. The self-attention mechanism refers to the Attention mechanism occurring between the elements in the Source or between the elements in the Target. It can also be understood as the attention mechanism of the special case of Target=Source. The specific calculation process is the same, only the calculation object changes.

[0173] (4) Natural language processing (NLP)

[0174] Natural language is human language, and natural language processing (NLP) is the processing of human language. Natural language processing is a process of systematic analysis, understanding and information extraction of text data in an intelligent and efficient way. By using NLP and its components, we can manage very large blocks of text data, or perform a large number of automated tasks, and solve a variety of problems, such as automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering and topic segmentation, etc.

[0175] (5) Pre-trained language model

[0176] A pre-trained language model is a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction tasks. Its training includes two stages. In the pre-training stage, the model is trained on large-scale unsupervised text for language model tasks, learning a word representation. In the fine-tuning stage, the model uses the parameters learned in the pre-training stage for initialization, and performs a few-step training on text classification, sequence labeling and other downstream tasks, so that the semantic information obtained by pre-training can be successfully transferred to downstream tasks.

[0177] (6) Inference / Deployment: The forward computation process of a neural network.

[0178] (7) Large language model (LLM): A large language model refers to a natural language processing model trained on large-scale data, usually with tens of billions or hundreds of billions of parameters. These models learn the general features of language through large amounts of text data in the pre-training phase, and can then be fine-tuned for specific tasks in downstream tasks.

[0179] (8) Cube: A unit in NPU for calculating matrix multiplication, which can only calculate the matrix multiplication of the same type of data, such as integer matrix multiplied by integer matrix, or floating-point matrix multiplied by floating-point matrix. Generally, the computing power of Cube in NPU chips is strong.

[0180] (9) Vector: A unit in NPU for calculating vector operations, which can perform a variety of vector operations, but the computing power is weaker than Cube.

[0181] (10) GM: Memory for storing data, with a large space, and the data involved in the calculation of Cube and Vector units needs to be read from GM, but the bandwidth (reading speed) of reading data from GM is small.

[0182] (11) L2: L2 cache (second-level cache) for storing data, with a relatively small space compared to GM, and a higher read-write speed compared to GM. At the beginning of the calculation process, the data is located in GM, which needs to be entered into L2 first, and then into Cube / UB; the data from Cube / UB can be directly entered into L2.

[0183] (12) L1: Level 1 cache, which is the cache in Cube, with a smaller space compared to L2, and a higher read-write speed compared to L2.

[0184] (13) L0A / L0B / L0C: Level 0 cache, which is the cache in Cube specifically used for storing calculation data, with the smallest space and the highest read-write speed. L0A / B / C is specifically used for storing left matrix / right matrix and calculation result matrix.

[0185] (14) Quantization: A method used for model compression and acceleration in large model inference, which compresses 16-bit weights into 8-bit for storage and subsequent calculation.

[0186] Large model inference accelerates the technology advancement of neural networks in academia and industry. Large model inference can extract valuable information from complex models and solve various problems such as image recognition, speech recognition, and natural language processing. However, the inference of large models usually requires a large amount of computing resources, which to some extent limits its widespread application and real-time performance. Therefore, how to accelerate the inference of large models and improve their efficiency in practical applications is a problem that needs to be solved.

[0187] Among them, the quantization and dequantization methods are essential methods for inference acceleration, and the core idea is as follows:

[0188] The parameters of the large model are generally stored by default in FP16 or BF16, and each parameter requires 16 bits of memory. Therefore, a quantization method is introduced to compress it into int8 format, and each parameter only requires 8 bits of memory, reducing the memory usage by half.

[0189] Referring to FIG. 6B, in the architecture of a graphics processing unit (GPU), tensor cores (C) and vectors (V) share a shared memory (so data transmission between C and V is very convenient), and the vector calculation unit is multiple, the computing power is relatively strong, and bf16 is supported. In the architecture of a neural processing unit (NPU), C and V are separated, and data from C to V (or from V to C) must pass through GM, the transmission speed is poor, and the cube calculation unit is multiple, and the computing power of the cube is relatively strong, the vector computing power is low, that is, the computing power of the vector is relatively poor, and bf16 format data calculation is not supported.

[0190] Based on the GPU and NPU architectures, the performance benefits of the same quantization operation are also different. Taking the A16W8 quantization format as an example, in some GPU architectures, quantization of the entire network has a performance benefit of 30%. However, the same quantization operation in some NPU architectures has a negative benefit of about 20% due to the unfriendly dequantization operator.

[0191] Specifically, first, the weight matrix in int8 format is moved into the vector calculation unit, and the dequantization operation is performed in the vector calculation unit to convert it to the fp16 / bf16 data type. Here, fp16 or bf16 is specifically taken, depending on the data type of the left matrix, that is, the data type of the weight matrix is changed to the same as the left matrix. Then, the fp16 / bf16 weight matrix is moved from the vector unit to the cube unit, and the matrix multiplication calculation in the fp16 / bf16 type is performed in the cube unit. It is noted that since the CV (between the cube unit and the vector unit) on the GPU has a shared SRAM storage unit, the weight data can be directly moved to the cube after quantization in the vector, and the moving of this part almost does not consume time. For the dequantization operator in incremental inference, the core time-consuming point comes from the moving of the weight matrix. Therefore, the calculation effect of this scheme is relatively good on the GPU.

[0192] The biggest drawback of the prior art is that it does not have an operator layer implementation scheme for NPU architecture. Directly reuse the GPU processing method, the performance in the CV-separated architecture of NPU is not good.

[0193] The core reason is that the weight matrix needs to be moved into a Vector once with the int8 data type (assuming that the row and column numbers of the weight matrix are K and N respectively, and the data amount is K*N, in bytes). After the inverse quantization calculation of the Vector, the weight matrix needs to be moved into a Cube with the fp16 / bf16 data type, and this step needs to be moved into the GM first, and then moved into the Cube from the GM. The amount of data moved from the GM to the Cube is 2KN, in bytes.

[0194] Therefore, the total amount of data read from the GM is 3KN for this scheme, compared with the scheme without using the quantization / inverse quantization algorithm (that is, the data is directly fp16 / bf16), the total amount of data read is 2KN. That is, the amount of data read by the quantization algorithm increases by 50%, so the performance of the quantization algorithm on the NPU not only does not improve, but also decreases. In the current actual data, the performance of the quantization operator is about 0.7-1 times that of the non-quantization operator.

[0195] On the NPU architecture in which the Cube and Vector calculation units are separated (without a high-speed data transmission channel), when implementing the inverse quantization operator, the weight matrix needs to be first transmitted to the Vector calculation unit, and after the inverse quantization calculation is completed, it is transmitted to the Cube calculation unit for subsequent calculation. The consequence is that the amount of data moved about the weight matrix is 3 times that of the ideal inverse quantization operator, and compared with the scheme without quantization operation, the amount of data moved is 1.5 times, so that the quantization operation not only does not have performance benefits, but also has performance loss.

[0196] To solve the above problems, an embodiment of the present application provides a data processing method. The data processing method of the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0197] Referring to FIG. 7, FIG. 7 is a flow diagram of a data processing method provided by an embodiment of the present application. As shown in FIG. 5, the data processing method provided by an embodiment of the present application can include steps 701-703, which will be described in detail below.

[0198] 701. Obtain a plurality of sub-values from a first calculation unit according to a first value, wherein different sub-values are digits at different orders of magnitude positions in the first value, the data type of the first value is different from the data type of a model parameter of a network layer, the data type of each sub-value is the same as the data type of the model parameter, and the model parameter is compressed data.

[0199] In the process of processing data by the machine learning model, the model parameters and the input data are often calculated, for example, the weights of the model and the input data are calculated, wherein the first value can be the input data, and the model parameters can be the compressed (for example, quantized) data, for example, the data type of the first value is floating point, and the data type of the model parameters is integer.

[0200] For example, the data type of the first value can be fp16, and the data type of the model parameters can be int8.

[0201] In the embodiments of the present application, in order to solve the problem of large memory reading caused by the transfer of a large amount of data in the memory after the dequantization operation, the baseline scheme of "decompressing the compressed data before use" is changed to the route of "processing directly in the compressed space". Specifically, the input data (not the model parameters) is transformed to obtain a plurality of sub-values consistent with the data type of the model parameters, and the model parameters and each sub-value are operated, and then the operation result is merged to obtain the output. Since the dequantization process is not involved, that is, the values are in the compressed space, therefore, the data amount of the first data unit and the second data unit when obtaining data is small (not the data after dequantization), thereby reducing the data transfer amount and improving the operation efficiency.

[0202] The data type here can be represented by an integer type and a number of bits, for example, int8, fp16, etc.

[0203] The first value in the embodiments of the present application can be an element in a matrix, and operations similar to the first value can be performed on each element in the matrix.

[0204] In one possible implementation, a plurality of sub-values can be obtained from the first value by the first calculation unit, wherein different sub-values are numbers in different orders of magnitude in the first value. And the combination of different sub-values is mathematically equivalent to the first value, for example, the first value is 2.3, the order of magnitude of 2 is the unit digit, and the order of magnitude of 3 is the first digit after the decimal point. One of the sub-values can be 2, and the other sub-value can be 3. The combination of the sub-values is mathematically equivalent to the first value (that is, 2+0.3=2.3). For example, the first value is 12.35, the order of magnitude of 1 is ten, the order of magnitude of 2 is the unit digit, the order of magnitude of 3 is the first digit after the decimal point, and the order of magnitude of 5 is the second digit after the decimal point. One of the sub-values can be 12, and the other sub-value can be 35. That is, different sub-values can correspond to one or more numbers of different orders of magnitude, and different sub-values correspond to numbers of different orders of magnitude, for example, "different orders of magnitude positions" can be understood as completely non-overlapping orders of magnitude.

[0205] In a possible implementation, the first value is of a floating point type, and the plurality of sub-values includes a first sub-value and a second sub-value; the first sub-value is a partial digit of an integer of the first value, and the second sub-value is a partial digit of a decimal of the first value. That is, the integer of the first value can be split into a plurality of sub-values for representation (i.e., multi-scale splitting of the integer), for example, the first value is 12.35, the first sub-value can be 1, and the second sub-value can be 2, and for another example, the first value is 125.35, the first sub-value can be 12, and the second sub-value can be 5.

[0206] In a possible implementation, the first value is of a floating point type, and the plurality of sub-values includes a first sub-value and a second sub-value; the first sub-value is a partial digit of an integer of the first value, and the second sub-value is a partial digit of a decimal of the first value. That is, the integer of the first value can be split into a plurality of sub-values for representation (i.e., multi-scale splitting of the integer), for example, the first value is 12.35, the first sub-value can be 1, and the second sub-value can be 2, and for another example, the first value is 125.35, the first sub-value can be 12, and the second sub-value can be 5.

[0207] In a possible implementation, the first value is of a floating point type, and the plurality of sub-values includes a first sub-value and a second sub-value; the first sub-value is a partial digit of an integer of the first value, and the second sub-value is a partial digit of a decimal of the first value. That is, the integer of the first value can be split into a plurality of sub-values for representation (i.e., multi-scale splitting of the integer), for example, the first value is 12.35, the first sub-value can be 1, and the second sub-value can be 2, and for another example, the first value is 125.35, the first sub-value can be 12, and the second sub-value can be 5.

[0208] Of course, the integer can be multi-scale split, and the whole of the decimal can be represented by one sub-value, or the decimal can be multi-scale split, and the whole of the integer can be represented by one sub-value.

[0209] In a possible implementation, each sub-value can be represented by the same data type as the model parameter.

[0210] In a possible implementation, the number of bits of the data type of the first value is greater than the number of bits of the data type of each sub-value.

[0211] For example, the first value can be fp16, and the model parameter can be int8, and each sub-value can be represented by the data type of int8

[0212] By performing the above operation on the elements of the matrix containing the first value (that is, the left matrix in matrix operations), the matrix can be split into multiple matrices. For example, a floating-point data type can be split into multiple matrices represented by int.

[0213] 702. The second calculation unit calculates the model parameters and each of the sub-values ​​respectively to obtain multiple calculation results;

[0214] The first computation unit can be a vector computation unit, and the second computation unit can be a matrix (cube) computation unit. When the model parameters and sub-values ​​are integers, the matrix computation unit performs integer calculations, making it possible to use int calculations for cubes. This allows for the simultaneous use of multiple int representations, and int calculations are more precise than FP16. Matrix calculations within the cube unit use pure int data types. Cube calculations offer higher precision and avoid deterministic issues. The splitting of the left matrix and the merging of results are performed in the vector unit. Only the calculations of the left matrix and output results are performed in the vector unit, significantly reducing the required vector calculations and data transfer between CVs.

[0215] Taking fp16 as an example, the left matrix can be split into multiple int8 data type matrices, as shown in Figure 8. The input left matrix is ​​first split into multiple int8 type matrices. After splitting, the int8 type matrices and the weight matrix W are moved into a Cube unit to calculate matrix multiplication. Finally, the result of the matrix multiplication is moved into a Vector unit to calculate the final output. Int type cube calculations are used instead of floating-point type cube calculations. That is, the method of first splitting the left matrix into ints, then multiplying the split left matrix with the weight matrix, and finally merging the results is a dequantization operator calculation method. This eliminates the need for any weight processing, achieving the theoretically optimal level of weight data transfer—the largest core time-consuming point.

[0216] In one possible implementation, the plurality of sub-values ​​and the model parameters can be moved from the memory to the second computing unit; and the plurality of calculation results obtained by the second computing unit can be moved to the memory.

[0217] In one possible implementation, the model parameters can be multiplied by each of the sub-values ​​using a second computing unit.

[0218] 703. The first calculation unit merges the multiple calculation results to obtain a second value, wherein the data type of the second value is the same as that of the first value.

[0219] In a possible implementation, the plurality of calculation results can be carried from the memory to the first calculation unit.

[0220] Referring to FIG. 9, the embodiments of the present application can be used in the inference scene of a large language model based on an NPU architecture. In this kind of scene, due to the sharp expansion of the parameters of the model (the model size), the lengthening of the inference sequence, and other reasons, it is generally necessary to have a quantization / dequantization operation. The whole flow of operator layer calculation and calling is shown in the above figure, and the core device includes:

[0221] S1. NPU architecture: carrying and calculation unit: vector-cube without pass-through;

[0222] S2. Large model calculation network flow graph containing quantization / dequantization operation;

[0223] S3. Generate a calculation network flow graph with a new dequantization calculation operator logic;

[0224] Referring to FIG. 10, FIG. 10 is a structural schematic of a data processing device provided by the embodiments of the present application. As shown in FIG. 10, the data processing device provided by the embodiments of the present application includes:

[0225] The first calculation unit 1001 is configured to obtain a plurality of sub-values from a first value, wherein different sub-values are digits located at different orders of magnitude in the first value, the data type of the first value is different from the data type of a model parameter of a network layer, the data type of each sub-value is the same as the data type of the model parameter, and the model parameter is compressed data.

[0226] The second calculation unit 1002 is configured to calculate the model parameter and each sub-value respectively to obtain a plurality of operation results.

[0227] The first calculation unit 1001 is further configured to combine the plurality of operation results to obtain a second value, and the data type of the second value is the same as the data type of the first value.

[0228] In a possible implementation, the data type of the first value is a floating point type, and the sub-value is an integer type.

[0229] In a possible implementation, the number of bits of the data type of the first value is greater than the number of bits of the data type of each sub-value.

[0230] In a possible implementation, the first value is a floating point type, and the plurality of sub-values include a first sub-value and a second sub-value, wherein,

[0231] The first sub-value is a partial digit of an integer of the first value, and the second sub-value is a partial digit of the integer of the first value; or

[0232] The first sub-value is a partial digit of a decimal of the first value, and the second sub-value is a partial digit of the decimal of the first value; or

[0233] The first sub-value is all digits of the decimal of the first value, or the second sub-value is all digits of the integer of the first value.

[0234] In a possible implementation, the apparatus further includes:

[0235] The transceiver is configured to transfer the first value from the memory to the first calculation unit.

[0236] The transceiver is configured to transfer the plurality of sub-values obtained by the first calculation unit to the memory.

[0237] The transceiver is configured to transfer the plurality of sub-values and the model parameter from the memory to the second calculation unit.

[0238] The transceiver is configured to transfer a plurality of calculation results obtained by the second calculation unit to the memory.

[0239] The transceiver is configured to transfer the plurality of calculation results from the memory to the first calculation unit.

[0240] In a possible implementation, the memory is a global memory (GM), the first calculation unit is a vector calculation unit, and the second calculation unit is a matrix calculation unit.

[0241] In a possible implementation, the second calculation unit is further configured to perform a multiplication operation on the model parameter and each of the sub-values.

[0242] Next, a terminal device provided by an embodiment of the present application is introduced. Referring to FIG. 11, FIG. 11 is a structural schematic diagram of a terminal device provided by an embodiment of the present application. The terminal device 1100 can specifically be a virtual reality (VR) device, a mobile phone, a tablet computer, a notebook computer, a smart wearable device, etc., which is not limited herein. Specifically, the terminal device 1100 includes a receiver 1101, a transmitter 1102, a processor 1103, and a memory 1104 (wherein the number of processors 1103 in the terminal device 1100 can be one or more, and one processor is taken as an example in FIG. 11). The processor 1103 can include an application processor 11031 and a communication processor 11032. In some embodiments of the present application, the receiver 1101, the transmitter 1102, the processor 1103, and the memory 1104 can be connected through a bus or other means.

[0243] The memory 1104 can include read-only memory and random access memory, and provide the processor 1103 with instructions and data. A portion of the memory 1104 can also include non-volatile random access memory (NVRAM). The memory 1104 stores processor and operating instructions, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the operating instructions can include various operating instructions for implementing various operations.

[0244] The processor 1103 controls the operation of the execution device. In a specific application, various components of the execution device are coupled together through a bus system, which can include a data bus, a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, all the buses are referred to as a bus system in the figure.

[0245] The method disclosed in the above embodiments of the present application can be applied in the processor 1103 or implemented by the processor 1103. The processor 1103 can be an integrated circuit chip with a signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 1103 or the instruction in the form of software. The processor 1103 mentioned above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 1103 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 1104, and the processor 1103 reads the information in the memory 1104, and combines the hardware to complete the steps of the method involving model training or model inference process.

[0246] The receiver 1101 can be configured to receive inputted digital or character information, and to generate signal input related to relevant settings and function control of the execution device. The transmitter 1102 can be configured to output digital or character information through the first interface; the transmitter 1102 can also be configured to send instructions to the disk group through the first interface to modify data in the disk group; and the transmitter 1102 can further include a display device such as a display screen.

[0247] The server provided in the embodiments of the present application can have great differences due to different configurations or performances, and can include one or more central processing units (CPUs) 1212 (for example, one or more processors) and a memory 1232, one or more storage media 1230 (for example, one or more mass storage devices) for storing application programs 1242 or data 1244. The memory 1232 and the storage media 1230 can be temporary storage or persistent storage. The programs stored in the storage media 1230 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the server. Further, the central processing unit 1212 can be configured to communicate with the storage media 1230 and execute the series of instruction operations in the storage media 1230 on the server 1200.

[0248] The server 1200 can further include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1258, or one or more operating systems 1241, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and the like.

[0249] In the embodiments of the present application, the central processing unit 1212 is configured to perform the actions related to model training or model inference in the above embodiments.

[0250] The embodiments of the present application further provide a computer program product, which, when running on a computer, causes the computer to perform the steps performed by the execution device as described above, or causes the computer to perform the steps performed by the training device as described above.

[0251] The embodiments of the present application further provide a computer readable storage medium, which stores a program for signal processing, and when running on a computer, causes the computer to perform the steps performed by the execution device as described above, or causes the computer to perform the steps performed by the training device as described above.

[0252] The execution device, the training device or the terminal device provided by the embodiments of the present application can be a chip, which includes a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the execution device executes the data processing method described in the above embodiments, or so that the chip in the training device executes the data processing method described in the above embodiments. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0253] Specifically, please refer to FIG. 13, which is a structural schematic diagram of a chip provided by the embodiments of the present application. The chip can be a neural network processor NPU 1300, which is mounted on a host CPU (Host CPU) as a coprocessor and is assigned tasks by the Host CPU. The core part of the NPU is an operation circuit 1303, which extracts matrix data in a memory and performs multiplication operation under the control of a controller 1304.

[0254] In some implementations, the operation circuit 1303 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 1303 is a two-dimensional systolic array. The operation circuit 1303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1303 is a general-purpose matrix processor.

[0255] For example, it is assumed that there are an input matrix A, a weight matrix B and an output matrix C. The operation circuit takes corresponding data of the matrix B from the weight memory 1302 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1301 and performs matrix operation with the matrix B, and the partial result or final result of the obtained matrix is saved in an accumulator 1308.

[0256] The unified memory 1306 is used to store input data and output data. The weight data is transferred into the weight memory 1302 through the Direct Memory Access Controller (DMAC) 1305. The input data is also transferred into the unified memory 1306 through the DMAC.

[0257] The BIU is the Bus Interface Unit 1310, which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1309.

[0258] The BIU 1310 is used for the instruction fetch buffer 1309 to fetch instructions from the external memory, and is also used for the memory access controller 1305 to fetch the original data of the input matrix A or the weight matrix B from the external memory.

[0259] The DMAC is mainly used to transfer the input data in the external memory DDR into the unified memory 1306, or to transfer the weight data into the weight memory 1302, or to transfer the input data into the input memory 1301.

[0260] The vector calculation unit 1307 includes a plurality of operation processing units, which further process the output of the operation circuit 1303 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / full connection layer network calculation in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0261] In some implementations, the vector calculation unit 1307 can store the processed output vector to the unified memory 1306. For example, the vector calculation unit 1307 can apply a linear function; or, a nonlinear function to the output of the operation circuit 1303, such as linear interpolation on the feature planes extracted by the convolutional layer, and further, a vector of accumulated values to generate activation values. In some implementations, the vector calculation unit 1307 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1303, such as for use in a subsequent layer in the neural network.

[0262] The controller 1304 is connected to the instruction fetch buffer 1309, which is used to store instructions used by the controller 1304;

[0263] The unified memory 1306, the input memory 1301, the weight memory 1302, and the instruction memory 1309 are on-chip memories. The external memory is private to the NPU hardware architecture.

[0264] Any processor mentioned in the above can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above programs.

[0265] In addition, it should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the apparatus embodiment provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0266] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits, or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, a training device, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0267] In the above embodiments, all or part can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part.

[0268] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A data processing method, characterized by, The method comprises: According to the first numerical value, a plurality of sub-numerical values are obtained by a first calculation unit, wherein different sub-numerical values are digits in the first numerical value located at different orders of magnitude positions, the data type of the first numerical value is different from the data type of a model parameter of a network layer, the data type of each sub-numerical value is the same as the data type of the model parameter, and the model parameter is compressed data; The model parameter is calculated with each sub-numerical value by a second calculation unit to obtain a plurality of operation results; The plurality of operation results are combined by the first calculation unit to obtain a second numerical value, and the data type of the second numerical value is the same as the data type of the first numerical value.

2. The method of claim 1, wherein, The method further comprises: The first numerical value is carried from a memory to the first calculation unit; The plurality of sub-numerical values obtained by the first calculation unit are carried to the memory; The plurality of sub-numerical values and the model parameter are carried from the memory to the second calculation unit; The plurality of operation results obtained by the second calculation unit are carried to the memory; The plurality of operation results are carried from the memory to the first calculation unit.

3. The method according to claim 1 or 2, characterized in that, The data type of the first numerical value is a floating point type, and the sub-numerical value is an integer type.

4. The method according to any one of claims 1 to 3, characterized in that, The number of bits of the data type of the first numerical value is greater than or equal to the number of bits of the data type of each sub-numerical value.

5. The method according to any one of claims 1 to 4, characterized in that, The first numerical value is a floating point type, and the plurality of sub-numerical values include a first sub-numerical value and a second sub-numerical value; wherein The first sub-numerical value is part of the digits of the integer of the first numerical value, and the second sub-numerical value is part of the digits of the integer of the first numerical value; or The first sub-numerical value is part of the digits of the decimal of the first numerical value, and the second sub-numerical value is part of the digits of the decimal of the first numerical value; or The first sub-numerical value is all digits of the decimal of the first numerical value, or the second sub-numerical value is all digits of the integer of the first numerical value.

6. The method according to any one of claims 1 to 5, characterized in that, The memory is a global memory GM, the first calculation unit is a vector calculation unit, and the second calculation unit is a matrix calculation unit.

7. The method according to any one of claims 1 to 6, characterized in that, The calculation of the model parameter with each sub-numerical value by the second calculation unit comprises: The model parameter is multiplied with each sub-numerical value by the second calculation unit.

8. A data processing apparatus, characterized by, The device comprises: A first calculation unit is configured to obtain a plurality of sub-numerical values from a first numerical value, wherein different sub-numerical values are digits in the first numerical value located at different orders of magnitude positions, the data type of the first numerical value is different from the data type of a model parameter of a network layer, the data type of each sub-numerical value is the same as the data type of the model parameter, and the model parameter is compressed data; A second calculation unit is configured to calculate the model parameter with each sub-numerical value to obtain a plurality of operation results; The first calculation unit is further configured to combine the plurality of operation results to obtain a second numerical value, and the data type of the second numerical value is the same as the data type of the first numerical value.

9. The apparatus of claim 8, wherein, The data type of the first numerical value is a floating point type, and the sub-numerical value is an integer type.

10. The apparatus of claim 8 or 9, wherein, The first numerical value has a data type with a number of bits greater than a number of bits of a data type of each of the sub-numerical values.

11. The apparatus of any one of claims 8 to 10, wherein, The first numerical value is of a floating point type, and the plurality of sub-numerical values includes a first sub-numerical value and a second sub-numerical value; wherein, The first sub-numerical value is a partial digit of an integer of the first numerical value, and the second sub-numerical value is a partial digit of the integer of the first numerical value; or, The first sub-numerical value is a partial digit of a decimal of the first numerical value, and the second sub-numerical value is a partial digit of the decimal of the first numerical value; or, The first sub-numerical value is a full digit of the decimal of the first numerical value, or the second sub-numerical value is a full digit of the integer of the first numerical value.

12. The apparatus of any one of claims 8 to 11, wherein, The apparatus further includes: a transceiving module configured to transfer the first numerical value from a memory to the first computing unit; a transferring module configured to transfer the plurality of sub-numerical values obtained by the first computing unit to the memory; a transferring module configured to transfer the plurality of sub-numerical values and the model parameter from the memory to the second computing unit; a transferring module configured to transfer the plurality of calculation results obtained by the second computing unit to the memory; a transferring module configured to transfer the plurality of calculation results from the memory to the first computing unit.

13. The apparatus of any one of claims 8 to 12, wherein, The memory is a global memory (GM), the first computing unit is a vector computing unit, and the second computing unit is a matrix computing unit.

14. The apparatus of any one of claims 8 to 13, wherein, The second computing unit is further configured to perform a multiplication operation on the model parameter and each of the sub-numerical values.

15. A computer storage medium, comprising, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform operations of the method of any one of claims 1-7.

16. A computer program product, characterised in that, A computer readable instruction is included, which, when executed on a computer device, causes the computer device to perform the method of any one of claims 1-7.

17. A system comprising at least one processor, at least one memory; the processor, the memory are connected through a communication bus and complete communication between each other; The at least one memory is configured to store code; The at least one processor is configured to execute the code to perform the method of any one of claims 1-7.

18. A chip comprising a processor, characterized in that The processor is configured to support a data processing apparatus to implement the method of any one of claims 1-7. The processor is configured to support a data processing apparatus to implement the method of any one of claims 1-7.

Images