Data processing method and device

Abstract

The embodiment of the invention discloses a data processing method and device, relates to the technical field of computers, and is used for realizing data sampling of target distribution. The method comprises the following steps: determining probability distribution of data in an initial data set in a plurality of intervals of the initial data set; constructing a proposal distribution function according to the probability distribution of the data in the initial data set in a plurality of intervals of the initial data set; determining a target offset according to the proposal distribution function and the target distribution function; and generating a target data set according to the target offset and the interval range of the initial data set. According to the scheme, on the basis of the characteristics of discrete and low-dimensional evaluation length distribution of a large model, a sampling rejection method is adopted, interval division is carried out on sampled data in an initial data set, probability distribution of the sampled data in the intervals of the initial data set is counted, and then the target offset is found by traversing the limited intervals, so that the evaluation accuracy is improved. And adding the probability density of the sample distribution and the target offset so as to realize data sampling of the target distribution.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to data processing methods and apparatus. Background Technology

[0002] Large models (such as large language models (LLMs)) refer to deep learning models with tens or even hundreds of millions of parameters. Large models contain neural network models with extremely large numbers of parameters (usually over a billion). These models are widely used in the field of natural language processing (NLP) and are gradually expanding to other fields, such as computer vision, audio processing, and multimodal data processing.

[0003] Currently, evaluation datasets for large models mainly focus on accuracy. In order to objectively evaluate the inference performance of large models, it is necessary to construct a benchmark dataset that covers input and output distributions of different lengths to evaluate the performance related to the model's generation capabilities. Summary of the Invention

[0004] This application provides a data processing method and apparatus for sampling data from a target distribution. To achieve the above objectives, this application adopts the following technical solution:

[0005] In a first aspect, embodiments of this application provide a data processing method, which includes: determining the probability distribution of data in an initial dataset across multiple intervals of the initial dataset; constructing a proposed distribution function based on the probability distribution of data in the initial dataset across multiple intervals of the initial dataset; determining a target offset based on the proposed distribution function and a target distribution function; and generating a target dataset based on the target offset and the interval range of the initial dataset.

[0006] The embodiments of this application provide a solution that, based on the discrete and low-dimensional characteristics of the length distribution of large model evaluation, adopts a rejection sampling method. First, the sampled data in the initial dataset is divided into intervals and the probability distribution of the sampled data in the intervals of the initial dataset is statistically analyzed. Then, by traversing these finite intervals, the maximum difference in probability density between the sampled sample and the target distribution function (i.e., the target offset) is found. The probability density of the sample distribution is added to this maximum difference instead of multiplying the probability density of the sample distribution by the K value in the conventional rejection sampling method. This also ensures that the probability density of the sample is greater than the current distribution density, and cleverly avoids the difficulty of finding the rejection sampling K value. Thus, the target dataset is generated with a smaller computational cost to achieve data sampling of the target distribution.

[0007] In one possible implementation, the initial dataset interval corresponding to any sample in the target dataset satisfies: bias + q(a) ≤ p(a), where bias is the target offset, q(x) is the proposed distribution function, p(x) is the target distribution function, and a is the initial dataset interval corresponding to any sample in the target dataset.

[0008] It is understandable that the implementation of rejection sampling is relatively simple. The method provided in this application adds a judgment condition of bias+q(a)≤p(a) to the sampling process to determine whether to accept or reject the sample. This method does not require complex mathematical derivation or advanced algorithms.

[0009] In one possible implementation, the target offset is the maximum difference between the proposed distribution function and the target distribution function.

[0010] The method provided in this application adds the probability density of the sample distribution to the maximum difference instead of multiplying the probability density of the sample distribution to the K value in conventional rejection sampling methods. This also ensures that the sample probability density is greater than the current distribution density, and cleverly avoids the difficulty of finding the rejection sampling K value.

[0011] In one possible implementation, the target distribution function is a Gaussian probability distribution function, a Zipf distribution function, a binomial distribution function, a Poisson distribution function, a geometric distribution function, a hypergeometric distribution function, a uniform distribution function, an exponential distribution function, or a normal distribution function.

[0012] It is understandable that the basic idea of ​​rejection sampling is to use an auxiliary distribution (i.e., a reference distribution) to generate samples, and to accept or reject these samples with a certain probability. The method provided in this application uses Gaussian probability distribution function, Zipf function, binomial distribution function, Poisson distribution function, geometric distribution function, hypergeometric distribution function, uniform distribution function, exponential distribution function, normal distribution function, etc. as auxiliary distributions to generate samples that conform to the target distribution.

[0013] Secondly, embodiments of this application provide a data processing apparatus. This apparatus can be an electronic device, a module applied to an electronic device (such as a processor, chip, or chip system), or a logic node, logic module, or software capable of implementing all or part of the functions of an electronic device. The apparatus includes a transceiver unit and a processing unit. The processing unit is configured to determine the probability distribution of data in an initial dataset across multiple intervals of the initial dataset. The processing unit is further configured to construct a proposed distribution function based on the probability distribution. The processing unit is further configured to determine a target offset based on the proposed distribution function and a target distribution function. The processing unit is further configured to generate a target dataset based on the target offset and the interval range of the initial dataset.

[0014] In one possible implementation, the initial dataset interval corresponding to any sample in the target dataset satisfies: bias + q(a) ≤ p(a), where bias is the target offset, q(x) is the proposed distribution function, p(x) is the target distribution function, and a is the initial dataset interval corresponding to any sample in the target dataset.

[0015] In one possible implementation, the target offset is the maximum difference between the proposed distribution function and the target distribution function.

[0016] In one possible implementation, the target distribution function is a Gaussian probability distribution function, a Zipf distribution function, a binomial distribution function, a Poisson distribution function, a geometric distribution function, a hypergeometric distribution function, a uniform distribution function, an exponential distribution function, or a normal distribution function.

[0017] Thirdly, embodiments of this application also provide a data processing apparatus, which includes at least one processor, which, when executing program code or instructions, implements the method described in the first aspect or any possible implementation thereof.

[0018] Alternatively, the data processing device may be a chip or a chip system.

[0019] Optionally, the device may further include at least one memory for storing the program code or instructions.

[0020] Fourthly, embodiments of this application also provide a chip, including: an input interface, an output interface, and at least one processor. Optionally, the chip further includes a memory. The at least one processor is used to execute code in the memory, and when the at least one processor executes the code, the chip implements the method described in the first aspect or any possible implementation thereof.

[0021] Alternatively, the chip described above can also be an integrated circuit.

[0022] Fifthly, embodiments of this application also provide a computer-readable storage medium for storing a computer program, the computer program including methods for implementing the first aspect or any possible implementation thereof.

[0023] Sixthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to implement the method described in the first aspect or any possible implementation thereof.

[0024] The data processing device, computer storage medium, computer program product, and chip provided in this embodiment are all used to execute the data processing method provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the data processing method provided above, and will not be repeated here. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0027] Figure 2 A schematic diagram of a probability distribution provided for an embodiment of this application;

[0028] Figure 3 A schematic diagram of another probability distribution provided for an embodiment of this application;

[0029] Figure 4 A schematic diagram of yet another probability distribution provided in an embodiment of this application;

[0030] Figure 5 A schematic diagram of yet another probability distribution provided in an embodiment of this application;

[0031] Figure 6 A schematic diagram of yet another probability distribution provided in an embodiment of this application;

[0032] Figure 7 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0033] Figure 8 This is a schematic diagram of the structure of a chip provided in an embodiment of this application;

[0034] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the embodiments of this application.

[0036] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0037] The terms "first" and "second," etc., in the specification and drawings of the embodiments of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0038] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0039] It should be noted that in the description of the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0040] The following explains the terminology used in the solutions provided in the embodiments of this application:

[0041] A benchmark dataset is a set of standardized test data used to evaluate the performance of computer algorithms, models, or systems. These datasets are typically carefully designed and collected for specific domains or tasks to ensure comprehensiveness and impartiality in the testing. Benchmark datasets are generally categorized into structured, semi-structured, and unstructured data. Structured data has a fixed format and fields, such as tables in a database; semi-structured data falls between the two, such as data in XML or JSON format; unstructured data does not have a fixed format, such as text files or images.

[0042] Rejection sampling, a Monte Carlo algorithm, is used to sample data from complex probability distributions. It selectively samples from a simple probability density distribution (such as a uniform or normal distribution), then discards samples that do not meet the requirements of a target probability distribution, and resamples until samples that meet the requirements are obtained. The principle of rejection sampling is to sample from a simple probability density distribution, then selectively discard some samples and resample according to the requirements of the target probability distribution, thereby simulating the sampling results under the new probability density function. This method is applicable to any distribution with density, but it performs poorly in high-dimensional spaces.

[0043] The Monte Carlo algorithm is a numerical computation method that utilizes random sampling, based on the law of large numbers. It solves various complex problems by simulating stochastic processes, and is particularly suitable for problems that are difficult or impossible to solve analytically. The basic idea of ​​the Monte Carlo algorithm is to approximate the expected value or probability by the average of random samples. It is commonly used to solve problems that are difficult to handle with traditional analytical methods, approximating mathematical expectations, integrals, or probability distributions by randomly generating samples in the problem space.

[0044] Currently, evaluation datasets for large models mainly focus on accuracy. In order to objectively evaluate the inference performance of large models, it is necessary to construct a benchmark dataset that covers input and output distributions of different lengths to evaluate the performance related to the model's generation capabilities.

[0045] Therefore, embodiments of this application provide a data processing method and apparatus for sampling data from a target distribution.

[0046] Figure 1 This application illustrates a data processing method provided by an embodiment of the present application, such as... Figure 1 As shown, the method includes:

[0047] S101. Determine the probability distribution of the data in the initial dataset across multiple intervals of the initial dataset.

[0048] For example, the initial dataset can be divided into multiple consecutive intervals of value ranges based on the size of the data in the initial dataset (input length or output length), and the probability distribution of the data in the initial dataset in multiple intervals of the initial dataset can be determined.

[0049] For example, the distribution of the six collected benchmark datasets can be used as the initial dataset for analysis. This initial dataset contains 24,189 data points, with a maximum value of 24,999 and a minimum value of 0. Therefore, the range of this initial dataset is 0 to 24,999. Assuming that this initial dataset is to be divided into 100 consecutive intervals, the numerical range and probability distribution corresponding to each interval are shown in Table 1.

[0050] As shown in Table 1, the numerical range of the first interval of the initial dataset is 0 to 249. The probability distribution of the data in the first interval of the initial dataset can be obtained by dividing the number of data points with values ​​between 0 and 249 by the total number of data points in the initial dataset. The numerical range of the second interval of the initial dataset is 250 to 499. The probability distribution of the data in the second interval of the initial dataset can be obtained by dividing the number of data points with values ​​between 250 and 499 by the total number of data points in the initial dataset. ... The numerical range of the 100th interval of the initial dataset is 24750 to 24999. The probability distribution of the data in the 100th interval of the initial dataset can be obtained by dividing the number of data points with values ​​between 24750 and 24999 by the total number of data points in the initial dataset.

[0051] Table 1

[0052]

[0053]

[0054] For example, if the initial dataset contains 24,189 data points, with a maximum value of 24,999 and a minimum value of 0, then the range of the initial dataset is 0 to 24,999. If we were to divide this initial dataset into 10 consecutive intervals, the numerical ranges and probability distributions corresponding to each interval would be shown in Table 2.

[0055] As shown in Table 2, the numerical range of the first interval of the initial dataset is 0 to 2499. The probability distribution of the data in the first interval of the initial dataset can be obtained by dividing the number of data points with values ​​between 0 and 2499 by the total number of data points in the initial dataset. The numerical range of the second interval of the initial dataset is 2500 to 4999. The probability distribution of the data in the second interval of the initial dataset can be obtained by dividing the number of data points with values ​​between 2500 and 4999 by the total number of data points in the initial dataset. ... The numerical range of the tenth interval of the initial dataset is 22500 to 24999. The probability distribution of the data in the tenth interval of the initial dataset can be obtained by dividing the number of data points with values ​​between 22500 and 24999 by the total number of data points in the initial dataset.

[0056] Table 2

[0057] probability distribution Interval 1 (0~2499) The number of data points with values ​​between 0 and 2499 in the initial dataset / the total number of data points in the initial dataset Interval 2 (2500~4999) The number of data points with values ​​between 2500 and 4999 in the initial dataset / the total number of data points in the initial dataset Interval 3 (5000~7499) The number of data points with values ​​between 5000 and 7499 in the initial dataset / the total number of data points in the initial dataset …… Interval 10 (22500~24999) The number of data points with values ​​between 22500 and 24999 in the initial dataset / the total number of data points in the initial dataset

[0058] For example, refer to Figure 2 The initial dataset has a maximum value of 25,000 and a minimum value of 0. The dataset is divided into 100 consecutive intervals. Figure 2 The left side shows the probability distribution of the data in the initial dataset over the range of 0 to 25000. Figure 2 The right side shows the probability distribution of the data in the initial dataset across 100 intervals of the initial dataset.

[0059] S102. Construct a proposed distribution function based on the probability distribution of the data in the initial dataset across multiple intervals of the initial dataset.

[0060] For example, according to Figure 2 The probability distribution of the data in the initial dataset shown on the right over 100 intervals of the initial dataset is used to construct the proposed distribution function q(x) corresponding to the initial dataset.

[0061] S103. Determine the target offset based on the proposed distribution function and the target distribution function.

[0062] The target distribution function can be a Gaussian probability distribution function, Zipf function, binomial distribution function, Poisson distribution function, geometric distribution function, hypergeometric distribution function, uniform distribution function, exponential distribution function, or normal distribution function.

[0063] For example, such as Figure 3As shown, the zipf function can be selected as the target distribution function p(x), and then the target bias can be determined based on the target distribution function p(x) and the proposed distribution function q(x).

[0064] For example, if the initial dataset is divided into 100 intervals, then the result of bias = argmax q(x) - zipf(x) can be calculated in the range of 1 to 100 (i.e., the value of x is in the range of 1 to 100) and used as the target offset.

[0065] For example, if the initial dataset is divided into 100 intervals, then the result of bias = max(q(x) - zipf(x)) can be calculated in the range of 1 to 100 and used as the target offset.

[0066] Where zipf(x) is the zipf function, zipf(x) = 1 / x.

[0067] It can be seen that the maximum difference between the proposed distribution function q(x) and the target distribution function (zipf(x)) can be calculated by argmax q(x)-zipf(x) or max(q(x)-zipf(x)).

[0068] For example, such as Figure 4 As shown, a Gaussian probability distribution function can be selected as the target distribution function p(x), and then the target offset bias can be determined based on the target distribution function p(x) and the proposed distribution function q(x).

[0069] For example, if the initial dataset is divided into 100 intervals, then within the range of 1 to 100, the result of bias = argmax q(x) - gaussian(x) can be calculated and used as the target offset.

[0070] For example, if the initial dataset is divided into 100 intervals, then the result of bias = max(q(x) - gaussian(x)) can be calculated in the range of 1 to 100 and used as the target offset.

[0071] Where gaussian(x) is the Gaussian probability distribution function. μ represents the expected value of the data, σ represents the standard deviation of the data, x is a random variable, and e is the natural constant.

[0072] It can be seen that the maximum difference between the proposed distribution function q(x) and the target distribution function (gaussian(x)) can be calculated by argmax q(x)-gaussian(x) or max(q(x)-gaussian(x)).

[0073] In one possible implementation, the target offset is the maximum difference between the proposed distribution function and the target distribution function.

[0074] For example, the target offset can satisfy: bias = argmax q(x) - p(x). Where bias is the target offset, q(x) is the above-mentioned proposal distribution function, and p(x) is the above-mentioned target distribution function.

[0075] For example, the target offset can satisfy: bias = max(q(x) - p(x)). Where bias is the target offset, q(x) is the proposed distribution function, and p(x) is the target distribution function.

[0076] The method provided in this application adds the probability density of the sample distribution to the maximum difference instead of multiplying the probability density of the sample distribution to the K value in conventional rejection sampling methods. This also ensures that the sample probability density is greater than the current distribution density, and cleverly avoids the difficulty of finding the rejection sampling K value.

[0077] S104. Generate the target dataset based on the target offset and the range of the initial dataset.

[0078] For example, samples can be generated uniformly from the range of the initial dataset. Preset conditions are set according to the target offset. Samples that do not meet the preset conditions are discarded, and samples that meet the preset conditions are retained. Samples are generated again, samples that do not meet the preset conditions are discarded, and samples that meet the preset conditions are retained, until the number of retained samples reaches a preset number.

[0079] For example, suppose the target dataset requires 1000 samples. The maximum value of the data in the initial dataset is 24999 and the minimum value is 0. Then the range of the initial dataset is 0 to 24999. The initial dataset is divided into 100 intervals. The value range of the first interval of the initial dataset is 0 to 249, the value range of the second interval is 250 to 499, ..., and the value range of the 100th interval is 24750 to 24999. Within the range of the initial dataset (0 to 24999), samples are generated uniformly. Suppose the value of the generated sample is 100. Since the value of this sample is within the value range of the first interval of the initial dataset (0 to 249), the corresponding interval of the initial dataset is 1. If bias + q(1) ≤ p(1), then the sample is retained and used as a sample in the target dataset. If bias + q(1) > p(1), then the sample is discarded. This process is repeated until 1000 samples are obtained.

[0080] In one possible implementation, the initial dataset interval corresponding to any sample in the target dataset can satisfy: bias + q(a) ≤ p(a), where bias is the target offset, q(x) is the proposed distribution function, p(x) is the target distribution function, and a is the initial dataset interval corresponding to any sample in the target dataset.

[0081] For example, such as Figure 5 As shown, assuming the zipf function is chosen as the target distribution function p(x), then the initial dataset interval corresponding to any sample in the target dataset can satisfy: bias + q(a) ≤ zipf(a).

[0082] For example, suppose the maximum value of the data in the initial dataset is 24999 and the minimum value is 0. The initial dataset is divided into 100 consecutive intervals. The value range of the first interval of the initial dataset is 0 to 249, the value range of the second interval is 250 to 499, ..., and the value range of the 100th interval is 24750 to 24999. The value of sample K in the target dataset is 300. Since the value range of the second interval of the initial dataset is 250 to 499, the interval of the initial dataset corresponding to sample K in the target dataset is 2. Since sample K is a sample in the target dataset, bias+q(2)≤zipf(2).

[0083] For example, such as Figure 6 As shown, assuming a Gaussian probability distribution function is chosen as the target distribution function p(x), then the initial dataset interval corresponding to any sample in the target dataset can satisfy: bias+q(a)≤gaussian(a).

[0084] For example, suppose the maximum value of the data in the initial dataset is 24999 and the minimum value is 0. The initial dataset is divided into 100 consecutive intervals. The value range of the first interval of the initial dataset is 0 to 249, the value range of the second interval is 250 to 499, ..., and the value range of the 100th interval is 24750 to 24999. The value of sample M in the target dataset is 24800. Since the value range of the 100th interval of the initial dataset is 24750 to 24999, the interval of the initial dataset corresponding to sample M in the target dataset is 100. Since sample M is a sample in the target dataset, bias+q(100)≤gaussian(100).

[0085] For example, a sample can be generated from the proposed distribution described above, its corresponding interval x can be found, and a number u can be uniformly generated from [0, Cq(x)]. If u ≤ p(x), this sample is accepted as a sample of the target distribution; otherwise, it is rejected. Where Cq(x) = bias + q(x).

[0086] For example, a number u can be uniformly generated from [0, bias+q(x)], and a number len can be uniformly generated from [min, max], with the result located in the interval v. If u ≤ zipf(v), then a sample of length len is accepted as a sample of the target distribution; otherwise, this sample is rejected, and resampling is performed until the required number is obtained. Here, max is the maximum value of the data in the initial dataset, and max is the minimum value of the data in the initial dataset.

[0087] For example, a number u can be uniformly generated from [0, bias+q(x)], and a number len can be uniformly generated from [min, max], and its interval v can be located. If u ≤ gaussian(v), then a sample of length len is accepted as a sample of the target distribution; otherwise, the sample is rejected, and resampling is performed until the required number is obtained.

[0088] The embodiments of this application provide a solution that, based on the discrete and low-dimensional characteristics of the length distribution of large model evaluation, adopts a rejection sampling method. First, the sampled data in the initial dataset is divided into intervals and the probability distribution of the sampled data in the intervals of the initial dataset is statistically analyzed. Then, by traversing these finite intervals, the maximum difference in probability density between the sampled sample and the target distribution function (i.e., the target offset) is found. The probability density of the sample distribution is added to this maximum difference instead of multiplying the probability density of the sample distribution by the K value in the conventional rejection sampling method. This also ensures that the probability density of the sample is greater than the current distribution density, and cleverly avoids the difficulty of finding the rejection sampling K value. Thus, the target dataset is generated with a smaller computational cost to achieve data sampling of the target distribution.

[0089] The following describes a data processing apparatus used to perform the above data processing methods.

[0090] It is understood that, in order to achieve the above-mentioned functions, the data processing device includes hardware and / or software modules corresponding to the execution of each function. Based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a manner that drives hardware or computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0091] This application embodiment can divide the data processing device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0092] Figure 7 The diagram illustrates a possible configuration of the data processing apparatus involved in the above embodiments. This apparatus can be an electronic device, a module applied to an electronic device (such as a processor, chip, or chip system), or a logic node, logic module, or software capable of implementing all or part of the functions of an electronic device. Figure 7 As shown, the data processing device 700 may include a transceiver unit 701 and a processing unit 702.

[0093] The aforementioned processing unit 702 is used to determine the probability distribution of data in the initial dataset across multiple intervals of the initial dataset.

[0094] The aforementioned processing unit 702 is also used to construct a proposed distribution function based on the aforementioned probability distribution.

[0095] The aforementioned processing unit 702 is further configured to determine the target offset based on the proposed distribution function and the target distribution function.

[0096] The aforementioned processing unit 702 is further configured to generate a target dataset based on the aforementioned target offset and the range of the aforementioned initial dataset.

[0097] In one possible implementation, the initial dataset interval corresponding to any sample in the target dataset satisfies: bias + q(a) ≤ p(a), where bias is the target offset, q(x) is the proposed distribution function, p(x) is the target distribution function, and a is the initial dataset interval corresponding to any sample in the target dataset.

[0098] In one possible implementation, the target offset is the maximum difference between the proposed distribution function and the target distribution function.

[0099] In one possible implementation, the target distribution function is a Gaussian probability distribution function, a Zipf distribution function, a binomial distribution function, a Poisson distribution function, a geometric distribution function, a hypergeometric distribution function, a uniform distribution function, an exponential distribution function, or a normal distribution function.

[0100] This application also provides a chip, which can be the chip of the data processing device described above. Figure 8 A schematic diagram of a chip 800 is shown. The chip 800 includes one or more processors 801 and interface circuitry 802. Optionally, the chip 800 may also include a bus 803.

[0101] The processor 801 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above data processing method can be completed through the integrated logic circuits in the processor 801 or through software instructions.

[0102] Optionally, the processor 801 described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0103] The interface circuit 802 can be used to send or receive data, instructions or information. The processor 801 can use the data, instructions or other information received by the interface circuit 802 to process the data, instructions or other information, and can send the processed information out through the interface circuit 802.

[0104] Optionally, the chip may also include memory, which may include read-only memory and random access memory, providing operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0105] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0106] Optionally, the chip can be used in the data processing apparatus or data processing device involved in the embodiments of this application. Optionally, the interface circuit 802 can be used to output the execution result of the processor 801. For the data processing methods provided by one or more embodiments of this application, please refer to the foregoing embodiments, which will not be repeated here.

[0107] It should be noted that the functions of the processor 801 and the interface circuit 802 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0108] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device can be a data processing device, a chip within the data processing device, or a functional module. For example... Figure 9 As shown, the electronic device 900 includes a processor 901, a transceiver 902, and a communication line 903.

[0109] The processor 901 is used to execute any step of the data processing method provided in the embodiments of this application, and during the execution of any step of the data processing method provided in the embodiments of this application, it may choose to call the transceiver 902 and the communication line 903 to complete the corresponding operation.

[0110] Furthermore, the electronic device 900 may also include a memory 904. The processor 901, memory 904, and transceiver 902 can be connected via a communication line 903.

[0111] The processor 901 can be a processor, a general-purpose processor, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 901 can also be other devices with processing capabilities, such as circuits, devices, or software modules, without limitation.

[0112] Transceiver 902 is used to communicate with other devices or other communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc. Transceiver 902 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0113] The transceiver 902 is mainly used for sending and receiving commands and information, and may include a transmitter and a receiver to send and receive commands and information, respectively; operations other than sending and receiving commands and information are implemented by the processor.

[0114] Communication line 903 is used to transmit information between the various components included in electronic device 900.

[0115] In one design, the processor can be viewed as a logic circuit, and the transceiver as an interface circuit.

[0116] Memory 904 is used to store instructions. These instructions can be computer programs.

[0117] The memory 904 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM). Memory 904 can also be a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0118] It should be noted that the memory 904 can exist independently of the processor 901, or it can be integrated with the processor 901. The memory 904 can be used to store instructions, program code, or some data, etc. The memory 904 can be located inside or outside the electronic device 900, without limitation. The processor 901 is used to execute the instructions stored in the memory 904 to implement the methods provided in the above embodiments of this application.

[0119] In one example, processor 901 may include one or more processor cores, for example Figure 9 The processor cores are 0 and 1.

[0120] As an optional implementation, the electronic device 900 includes multiple processors, for example, besides Figure 9 In addition to processor 901, it may also include processor 907.

[0121] As an optional implementation, the electronic device 900 also includes an output device 905 and an input device 906. For example, the input device 906 is a device such as a keyboard, mouse, microphone, or joystick, and the output device 905 is a device such as a display screen or speaker.

[0122] It should be noted that the electronic device 900 can be a chip system or... Figure 9 Devices with similar structures. The chip system can be composed of chips or include chips and other discrete components. Actions, terminology, etc., involved in the various embodiments of this application can be referenced interchangeably without limitation. The message names or parameter names in the messages used for interaction between devices in the embodiments of this application are merely examples; other names can be used in specific implementations without limitation. Furthermore, Figure 9 The structural composition shown does not constitute a limitation on the electronic device 900, except... Figure 9 In addition to the components shown, the electronic device 900 may include more than Figure 9 This may indicate more or fewer components, or a combination of certain components, or a different arrangement of components.

[0123] The processor and transceiver described in this application can be implemented on integrated circuits (ICs), analog ICs, radio frequency integrated circuits, mixed-signal ICs, application-specific integrated circuits (ASICs), printed circuit boards (PCBs), electronic devices, etc. The processor and transceiver can also be manufactured using various IC process technologies, such as complementary metal-oxide semiconductors (CMOS), n-metal-oxide-semiconductor (NMOS), positive-channel metal-oxide semiconductors (PMOS), bipolar junction transistors (BJTs), bipolar CMOS (BiCMOS), silicon germanium (SiGe), gallium arsenide (GaAs), etc.

[0124] This application also provides a data processing apparatus, which includes at least one processor. When the at least one processor executes program code or instructions, it implements the data processing method described above.

[0125] Optionally, the device may further include at least one memory for storing the program code or instructions.

[0126] This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on a data processing device, the data processing device performs the aforementioned related method steps to implement the data processing method described above.

[0127] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the data processing method described in the above embodiments.

[0128] This application also provides a data processing apparatus, which may specifically be a chip, integrated circuit, component, or module. Specifically, the apparatus may include a connected processor and a memory for storing instructions, or the apparatus may include at least one processor for fetching instructions from external memory. When the apparatus is running, the processor can execute the instructions to cause the chip to perform the data processing methods described in the above method embodiments.

[0129] It should be understood that in various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0130] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0131] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0133] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0135] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this application embodiment, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be included within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A data processing method, characterized in that, include: Determine the probability distribution of the data in the initial dataset across multiple intervals of the initial dataset; Construct a proposed distribution function based on the probability distribution; The target offset is determined based on the proposed distribution function and the target distribution function; The target dataset is generated based on the target offset and the range of the initial dataset.

2. The method according to claim 1, characterized in that, The initial dataset interval corresponding to any sample in the target dataset satisfies: bias+q(a)≤p(a), where bias is the target offset, q(x) is the proposed distribution function, p(x) is the target distribution function, and a is the interval of the initial dataset corresponding to any sample in the target dataset.

3. The method according to claim 1 or 2, characterized in that, The target offset is the maximum difference between the proposed distribution function and the target distribution function.

4. The method according to any one of claims 1 to 3, characterized in that, The target distribution function is a Gaussian probability distribution function, Zipf distribution function, binomial distribution function, Poisson distribution function, geometric distribution function, hypergeometric distribution function, uniform distribution function, exponential distribution function, or normal distribution function.

5. A data processing apparatus, characterized in that, include: Transceiver unit and processing unit; The processing unit is used to determine the probability distribution of data in the initial dataset across multiple intervals of the initial dataset; The processing unit is further configured to construct a proposal distribution function based on the probability distribution; The processing unit is further configured to determine the target offset based on the proposed distribution function and the target distribution function; The processing unit is further configured to generate a target dataset based on the target offset and the range of the initial dataset.

6. The apparatus according to claim 5, characterized in that, The initial dataset interval corresponding to any sample in the target dataset satisfies: bias+q(a)≤p(a), where bias is the target offset, q(x) is the proposed distribution function, p(x) is the target distribution function, and a is the initial dataset interval corresponding to any sample in the target dataset.

7. The apparatus according to claim 5 or 6, characterized in that, The target offset is the maximum difference between the proposed distribution function and the target distribution function.

8. The apparatus according to any one of claims 5 to 7, characterized in that, The target distribution function is a Gaussian probability distribution function, Zip distribution function, binomial distribution function, Poisson distribution function, geometric distribution function, hypergeometric distribution function, uniform distribution function, exponential distribution function, or normal distribution function.

9. A data processing apparatus, comprising at least one processor and a memory, characterized in that, The at least one processor executes a program or instructions stored in a memory to cause the data processing device to implement the method of any one of claims 1 to 4.

10. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is run on a computer or processor, it causes the computer or processor to perform the method of any one of claims 1 to 4.

11. A computer program product, the computer program product comprising instructions, characterized in that, When the instructions are executed on a computer or processor, the computer or processor performs the method of any one of claims 1 to 4.

Images