A data processing method and device based on machine learning

Abstract

An embodiment of the present invention provides a machine learning-based data processing method and apparatus, addressing the difficulty of implementing a self-attention mechanism in a SoftMax module. The method includes receiving a numerical sequence, obtaining an index sequence based on the number of objects in the numerical sequence, performing a weighted query on a real-world lookup table based on the index sequence to obtain a probability sequence, and configuring a machine learning system based on the probability sequence.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a data processing method and device based on machine learning. Background Art

[0002] In the past two years, the attention model has been widely used in various types of deep learning tasks, such as natural language processing, image recognition, and speech recognition. It is one of the core technologies in deep learning that deserves the most attention and in-depth understanding.

[0003] Because the SoftMax module of the attention model weights certain features more highly than others, allowing the entire system to focus on specific areas of the image, normalization in the SoftMax module is essential. If the normalization of the SoftMax module is removed, SoftMax no longer constitutes an attention mechanism and only represents a dot product of the features. In the existing SoftMax module, a pre-calculated division is required to implement the self-attention mechanism. This increases the computational difficulty and time when used intensively, making it difficult to implement the transformer module and self-attention mechanism in FPGAs. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a data processing method and device based on machine learning, which solves the problem that it is difficult for the SoftMax module to implement the self-attention mechanism.

[0005] An embodiment of the present invention provides a data processing method based on machine learning, including: receiving a digital sequence, obtaining an index sequence based on the number of objects in the digital sequence; performing a weighted query in a real query table based on the index sequence to obtain a probability sequence; and configuring a machine learning system based on the probability sequence.

[0006] In one embodiment, the digital sequence includes a preset number of numbers; the step of obtaining an index sequence based on the number of objects in the digital sequence includes: obtaining the preset number of integers based on the digital sequence; and arranging the preset number of integers according to a first preset rule to obtain the index series.

[0007] In one embodiment, the step of performing a weight query in a real query table based on the index sequence to obtain a probability sequence includes: obtaining the weight of each object in the index sequence from the real query table based on the index sequence; and sorting according to a second preset rule based on the weight of each object to obtain the probability sequence.

[0008] In one embodiment, the data in the actual query table are floating-point values; the sum of all the data in the actual query table is equal to 1.

[0009] In one embodiment, the data in the actual query table are integer values, and all data in the actual query table are powers of 2.

[0010] In one embodiment, the reality lookup table includes fixed values ​​and predetermined values.

[0011] A configuration device for a machine learning system includes: a receiving module for receiving a digital sequence; a processing module for obtaining an index sequence based on the number of objects in the digital sequence; performing a weighted query in a real query table based on the index sequence to obtain a probability sequence; and a configuration module for configuring the machine learning system based on the probability sequence.

[0012] In one embodiment, the digital sequence includes a preset number of numbers; the processing module is further used to: obtain a number of integers equal to the preset number based on the digital sequence; arrange the preset number of integers according to a first preset rule to obtain the index series; obtain the weight of each object in the index sequence from the real query table based on the index sequence; and sort according to a second preset rule based on the weight of each object to obtain the probability sequence.

[0013] An electronic device includes a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the above-mentioned machine learning-based data processing method.

[0014] A computer-readable storage medium stores a computer program, which, when executed by a processor, is used to implement the above-mentioned data processing method based on machine learning.

[0015] Embodiments of the present invention provide a machine learning-based data processing method and apparatus. The method receives a numerical sequence, obtains an index sequence based on the number of objects in the numerical sequence, performs a weighted query on a real-world lookup table based on the index sequence to obtain a probability sequence, and configures a machine learning system based on the probability sequence. The machine learning-based data processing method eliminates the division operation required for normalization when applying the SoftMax module in self-attention mechanisms and transformer modules, and allows the application of self-attention mechanisms and transformer modules without division operations, thereby supporting very high data throughput in applications such as image and video processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure shows a structural diagram of a convolutional neural network in the prior art.

[0017] Figure 2 The figure shows a schematic diagram of the input and output of a SoftMax module in the prior art.

[0018] Figure 3 Shown is a flowchart of a data processing method based on machine learning provided by one embodiment of the present invention.

[0019] Figure 4 FIG2 is a flowchart of a method for obtaining an index sequence provided by an embodiment of the present invention.

[0020] Figure 5 FIG2 is a flow chart of a method for obtaining a probability sequence provided by an embodiment of the present invention.

[0021] Figure 6 FIG. 1 is a schematic diagram of input and output of a hierarchical SoftMax module provided by an embodiment of the present invention.

[0022] Figure 7 FIG2 is a schematic diagram showing the structure of a machine learning device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0024] A convolutional neural network, or convnet for short, is a neural network architecture that uses images as input / output and replaces scalar weights with filters (convolutions). As an example, Figure 1 A simple structure with 3 layers is shown. The structure takes 4 input images on the left, has 3 units in the hidden layer in the middle (output images), and has 2 units in the output layer, generating 2 output images. Each weight is The box corresponds to a filter (e.g. a 3x3 or 5x5 kernel), where k is the label representing the input layer number, and i and j are the labels representing the input and output units, respectively. A scalar is added to the output of a convolution. The result of adding multiple convolutions and biases is then passed through an activation function, which often corresponds to a rectified linear unit (ReLU), a sigmoid function, or a hyperbolic tangent. The filters and biases are fixed during system operation, obtained through a training process using a set of input / output example images, and adjusted to meet certain optimization criteria based on the application. Typical configurations include one-tenth to several hundred filters per layer. Networks with three layers are generally considered shallow, while networks with more than five or ten layers are generally considered deep.

[0025] The self-attention mechanism can be used as part of the "Transformer" module; the basic idea of ​​the "Transformer" module is to use 3 inputs: query (Q), key (K) and value (V), and the attention function can be described as mapping the query and a set of key-value pairs to the output, where the query, key, value and output are all vectors, and the output is calculated as a weighted sum of the values, where the weight assigned to each value is calculated by a compatibility function of the query and the corresponding key.

[0026] A basic module of the self-attention mechanism in the transformer module is the SoftMax module, such as Figure 2 As shown, the module calculates the exponential function of each input feature and normalizes the output so that the sum of all features is equal to 1, where the input includes a sequence of N numbers, and N numbers are x1 to x n , the output probability series includes N objects, namely p1~pn; among them, Some computer vision applications of attention models, such as image classification, require very few division operations to solve the problem. This is because the algorithm reduces the image resolution several times before performing the division operation. Other applications, such as image restoration and enhancement, require a division operation for each output pixel in a large image. In such applications, the computational resources required by the SoftMax module are prohibitive.

[0027] The present invention replaces the traditional SoftMax module with a hierarchical SoftMax module. It receives a sequence of N numbers, applies the sorted module, obtains corresponding weights from a real-world lookup table based on the ranking index, and outputs a probability sequence of the same N numbers in order. This eliminates the division operation in the SoftMax module and allows the configuration of self-attention mechanisms and transformer modules without performing division calculations, thereby supporting very high data throughput in applications such as image and video processing. Specific implementations are described in the following examples.

[0028] This embodiment provides a data processing method based on machine learning, such as Figure 3As shown, the data processing method based on machine learning includes:

[0029] Step 01: Receive a digital sequence and obtain an index sequence based on the number of objects in the digital sequence.

[0030] The digital sequence includes a preset number of numbers, wherein the preset number is N integers, where N≥0.

[0031] The step of obtaining an index sequence based on a digital sequence includes: obtaining a preset number of integers based on the digital sequence; arranging the preset number of integers according to a first preset rule to obtain an index series. For example, a sequence of N numbers is received, and N integers are output, where these integers are arranged in descending order of value to form an index sequence. Figure 4 As shown, a digital sequence of N integers is input into the RANK (rank) module, and the digital sequence of N integers is {x1, x2, ... x n}; Output index sequence {r1, r2, ... r n}, where r i ∈{1,...,N},r i ≠r j If i≠j,

[0032] Step 02: Based on the index sequence, perform a weight query in the actual query table to obtain a probability sequence.

[0033] The step of performing a weight query in a real query table based on the index sequence to obtain a probability sequence includes: obtaining the weight of each object in the index sequence from the real query table based on the index sequence; and sorting the weight of each object according to a second preset rule to obtain a probability sequence.

[0034] Optionally, the actual query table contains fixed values ​​and predetermined values. The fixed values ​​are input values, and the predetermined values ​​are output values. The corresponding predetermined values ​​are searched in the query table according to the input values ​​for output.

[0035] The data in the actual query table are in floating point format. For the floating point format, the sum of all the values ​​in the actual query table is equal to 1.

[0036] The data in the query table is in integer format. For integer format, the sum of all the values ​​in the query table should be a constant, usually a power of 2.

[0037] like Figure 5 As shown, the LUT module is a real query table, and the weight of each object in the index sequence obtained in the real query table is LUT[1]~LUT[N], where C is the sum of LUT[1]~LUT[N]; LUT[1]~LUT[N] and the index sequence {r1,r2,……r n} Input the reorder module and output the probability sequence {p1, p2, ... p n}, where p i =LUT[r i ].

[0038] If one feature input to the SoftMax module is much larger than the inputs of the remaining features, the probability of that feature will be equal to 1 and the probabilities of the remaining features will be equal to 0. In this case, the output of SoftMax is equal to the maximum value of all inputs. Typically, the SoftMax module prefers the maximum value, but is "softer" than the maximum function. In the present invention, the weights in the LUT (realistic lookup table) represent a fixed configuration of softness levels, and the weights give priority to the maximum value. The weights in the LUT can express different degrees of smoothness, but when the highest weight included in the LUT is equal to 1 and the remaining weights are equal to 0, the weights in the LUT can also express the maximum function.

[0039] By querying the real lookup table, division by the constant can be achieved through a simple bit shift operation. Compared with the existing technology, the pre-calculated division in the SoftMax module is omitted, thereby reducing the calculation difficulty and time.

[0040] Step 03: Configure the machine learning system based on the probability sequence.

[0041] like Figure 6 As shown, the RANK module, LUT and reorder are combined into a hierarchical SoftMax module. When the digital sequence of N integers {x1, x2, ... x n After inputting the hierarchical SoftMax module, the probability sequence {p1, p2, ... p n}, where p i =LUT[r i ]. Among them, p in the probability sequence i Represents the probability used to multiply another sequence of N features in the self-attention mechanism.

[0042] When using the floating-point version, the multiplication in the hierarchical SoftMax module is the same as the multiplication in the existing SoftMax module. However, when using the integer version, the constant C (the sum of the values ​​in the actual lookup table) should be set to a power of 2. For example, if C = 2L, then each multiplication should be shifted by L. For example, if in the floating-point version, we multiply a*pi, then in the integer version, we calculate (a*pi) < <L。

[0043] This embodiment provides a configuration device 100 for a machine learning system, such as Figure 7 As shown, the machine learning configuration device 100 includes: a receiving module 10, a processing module 20 and a configuration module 30.

[0044] The receiving module 10 is used to receive a digital sequence and obtain an index sequence based on the number of objects in the digital sequence;

[0045] The processing module 20 is used to perform a weight query in the actual query table based on the index sequence to obtain a probability sequence;

[0046] The configuration module 30 is used to configure the machine learning system based on the probability sequence.

[0047] In addition, the digital sequence includes a preset number of numbers; the processing module 20 is further configured to:

[0048] Obtaining a number of integers equal to the preset number based on the digital sequence;

[0049] Arranging the preset number of integers according to a first preset rule to obtain the index series;

[0050] Obtaining a weight of each object in the index sequence from the actual query table based on the index sequence;

[0051] The probability sequence is obtained by sorting the objects according to a second preset rule based on the weight of each object.

[0052] This embodiment provides an electronic device. This embodiment provides an electronic device that may include a memory and a processor. The memory stores a computer program that, when executed by the processor, implements the machine learning-based data processing method described in the above embodiments. It is understood that the electronic device may also include an input / output (I / O) interface and a communication component.

[0053] The processor is used to execute all or part of the steps in the data processing method based on machine learning in the embodiment. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as data related to the application.

[0054] The processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and is used to execute the data processing method based on machine learning in the above embodiment.

[0055] The memory can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0056] This embodiment also provides a computer-readable storage medium. The functional units in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. If the functions are implemented as software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium.

[0057] Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention.

[0058] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, server, APP application store, and other media that can store program check codes, on which a computer program is stored. When the computer program is executed by a processor, the following method steps can be implemented:

[0059] Step 01: Receive a digital sequence and obtain an index sequence based on the number of objects in the digital sequence;

[0060] Step 02: Perform a weighted query in the actual query table based on the index sequence to obtain a probability sequence;

[0061] Step 03: Configure the machine learning system based on the probability sequence. The specific implementation methods and the effects produced can be referred to in the above embodiments, and the present invention will not be repeated here.

[0062] Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0063] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. On the other hand, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0064] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0065] The block diagrams of devices, apparatuses, equipment, and systems referred to in this application are intended only as illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems may be connected, arranged, or configured in any manner.

[0066] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0067] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0068] In the description of the application, the meaning of "multiple" is at least two, for example two, three, etc., unless otherwise clearly and specifically limited. In the embodiments of the present application, all directional indications (such as up, down, left, right, front, back, top, bottom ...) are only used to explain the relative position relationship, motion situation, etc. between each component under a certain specific posture (as shown in the drawings). If this specific posture changes, this directional indication also changes accordingly. In addition, the terms "comprise" and "have" and any deformation thereof are intended to cover non-exclusive inclusion. For example, the process, method, system, product or equipment comprising a series of steps or units is not limited to the steps or units listed, but optionally also includes the steps or units not listed, or optionally also includes other steps or units inherent to these processes, methods, products or equipment.

[0069] In addition, references to "embodiments" herein mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of such phrases in various places in the specification does not necessarily refer to the same embodiment, nor does it necessarily refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0070] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art who can easily think of changes or replacements within the technical scope disclosed by the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing method based on machine learning, characterized in that: include: receiving a digital sequence and obtaining an index sequence based on the number of objects in the digital sequence; Obtaining a weight of each object in the index sequence by querying from the LUT based on the index sequence, and sorting the objects based on the weight of each object to obtain a probability sequence; A machine learning system is configured based on the probability sequence. The machine learning system includes a hierarchical SoftMax module configured based on a level module, a LUT, and a rearrangement module for implementing a self-attention mechanism in a transformer module. The level module is used to receive a digital sequence and obtain an index sequence based on the number of objects in the digital sequence. After the digital sequence is input into the hierarchical SoftMax module, the weight of each object in the index sequence can be obtained by querying the LUT based on the index sequence, and the weight of each object is sorted to output the probability sequence.

2. The data processing method based on machine learning according to claim 1, characterized in that: The digital sequence includes a preset number of numbers; and the step of obtaining an index sequence based on the number of objects in the digital sequence includes: Obtaining the preset number of integers based on the digital sequence; The preset number of integers are arranged according to a first preset rule to obtain the index sequence.

3. The data processing method based on machine learning according to claim 1, characterized in that: The probability sequence is obtained by sorting the objects according to a second preset rule based on the weight of each object.

4. The data processing method based on machine learning according to claim 1, characterized in that: The data in the LUT is a floating point format value; the sum of all the data in the LUT is equal to 1.

5. The data processing method based on machine learning according to claim 1, characterized in that: The data in the LUT are integer format values, and all the data in the LUT are powers of 2.

6. The data processing method based on machine learning according to claim 1, characterized in that: The LUT includes fixed values ​​and predetermined values.

7. A configuration device for a machine learning system, characterized in that: include: A receiving module, configured to receive a digital sequence; a processing module, configured to obtain an index sequence based on the number of objects in the digital sequence; Obtaining a weight of each object in the index sequence by querying from the LUT based on the index sequence, and sorting the objects based on the weight of each object to obtain a probability sequence; A configuration module is used to configure a machine learning system based on the probability sequence, wherein the machine learning system includes a hierarchical SoftMax module configured based on a level module, a LUT, and a rearrangement module for implementing a self-attention mechanism in a transformer module, wherein the level module is used to receive a digital sequence and obtain an index sequence based on the number of objects in the digital sequence. After the digital sequence is input into the hierarchical SoftMax module, the weight of each object in the index sequence can be obtained by querying the LUT based on the index sequence, and the weight of each object is sorted to output the probability sequence.

8. The configuration device of the machine learning system according to claim 7, characterized in that: The digital sequence includes a preset number of numbers; the processing module is further configured to: Obtaining a number of integers equal to the preset number based on the digital sequence; Arranging the preset number of integers according to a first preset rule to obtain the index sequence; The probability sequence is obtained by sorting the objects according to a second preset rule based on the weight of each object.

9. An electronic device, characterized in that: It includes a memory and a processor, the memory is used to store one or more computer instructions, wherein when the one or more computer instructions are executed by the processor, the data processing method based on machine learning as described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, is used to implement the data processing method based on machine learning as described in any one of claims 1 to 6.

Images