Data processing method, data processing device, storage medium, and program product

Abstract

This application provides a data processing method, a data processing apparatus, a storage medium, and a program product. The method includes: acquiring a data sequence to be processed; wherein the data sequence includes multiple data items to be processed; determining header data and a difference sequence based on the data sequence; wherein the header data is the first data item to be processed in the data sequence; the difference sequence includes multiple differences, each difference being the difference between two adjacent data items; determining the encoding value corresponding to each difference based on the frequency of occurrence of the differences falling within a numerical range; wherein the frequency of occurrence is the number of differences falling within each numerical range; the encoding value is used to indicate the number of bits stored in the difference, and the number of bits in the encoding value is negatively correlated with the frequency of occurrence; and constructing a target data sequence based on the header data, the differences, and the encoding values ​​corresponding to the differences. This method is used to improve data compression rate and reduce the space occupied by data storage.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, a data processing device, a storage medium, and a program product. Background Technology

[0002] During data acquisition and processing, analog data collected by relevant devices can be converted into digital data for storage. Taking optical equipment as an example, many fields (such as biology, medicine, astronomy, physics, and semiconductors) utilize numerous analog-to-digital converter (ADC) modules in their optical devices (e.g., biological microscopes, telescopes, bright-field or dark-field defect detection equipment). This requires the storage and transmission capabilities of hardware such as field-programmable gate arrays (FPGAs) and microcontroller units (MCUs). The storage and transmission of this type of data is crucial to the performance of optical equipment. Existing data compression methods have low compression rates, resulting in large data storage space requirements and low transmission efficiency.

[0003] Therefore, there is an urgent need for a solution that can improve the data compression rate. Summary of the Invention

[0004] This application discloses a data processing method, a data processing device, a storage medium, and a program product for improving data compression rate.

[0005] Firstly, this application provides a data processing method, including:

[0006] Obtain the data sequence to be processed; wherein, the data sequence to be processed includes multiple data items to be processed;

[0007] Based on the data sequence to be processed, determine the header data and the difference sequence; wherein, the header data is the first data to be processed in the data sequence to be processed; the difference sequence includes multiple differences, each of which is the difference between two adjacent data to be processed in the multiple data to be processed;

[0008] The encoding value corresponding to the difference is determined based on the frequency of occurrence of the numerical range into which the difference falls; where the frequency of occurrence is the number of differences that fall into each numerical range; the encoding value is used to indicate the number of bits stored for the difference, and the number of bits of the encoding value is negatively correlated with the frequency of occurrence;

[0009] Construct the target data sequence based on the header data, the difference, and the corresponding encoded value of the difference.

[0010] Based on the above scheme, in some cases (especially when the data sequence to be processed consists of small fluctuations with little difference between consecutive data points), subtraction can transform the original data into a difference sequence with smaller values, requiring fewer bits for storage and thus being easier to compress. Furthermore, by preserving the header data to ensure the integrity of the difference sequence, a baseline value is provided for subsequent decompression. Finally, the encoded value indicates the number of bits required to store the difference, and the target data sequence is constructed based on the header data, the difference, and the encoded value. This facilitates understanding how to parse the data when parsing the target data sequence. Moreover, since the encoded value of the difference is negatively correlated with its frequency of occurrence, frequently occurring differences can be encoded with fewer bits, improving the compression ratio, thereby reducing the space occupied by data storage and improving data transmission efficiency.

[0011] In one possible implementation, the encoded value corresponding to the difference is determined based on the frequency of occurrences of the numerical range within which the difference falls, including:

[0012] Based on multiple differences, determine the frequency of occurrence of different numerical ranges;

[0013] Based on the frequency of occurrence of each numerical range, different numerical ranges are grouped to obtain N numerical range groups; where each numerical range group includes at least one numerical range, and N is an integer greater than 1.

[0014] For differences that fall within the numerical range, set the encoding value corresponding to the numerical range group to which the numerical range belongs, so as to obtain the encoding value corresponding to the difference.

[0015] Based on the above scheme, by grouping the numerical ranges and assigning the difference value to the corresponding coded value of the numerical range group, redundant storage space occupation caused by multiple encodings of the numerical range can be avoided.

[0016] In one possible implementation, before determining the frequency of occurrence of different numerical ranges based on multiple differences, the method further includes:

[0017] Determine the total number of occurrences for each value range group. The total number of occurrences is the sum of the number of occurrences of each value range included in the value range group.

[0018] The encoding value corresponding to the numerical range group is determined based on the total number of occurrences of the numerical range group; wherein, the encoding value corresponding to the numerical range group indicates the maximum number of bits required for storage in the numerical range group; the number of bits of the encoding value increases as the total number of occurrences of the numerical range group decreases.

[0019] Based on the above scheme, a frequency-based grouping strategy is used to group numerical ranges, and dynamic encoding optimization is performed based on these range groups. This allows for assigning shorter-bit encoding values ​​to differences that occur more frequently within a numerical range, achieving data compression and reducing storage redundancy.

[0020] In one possible implementation, the different numerical ranges are grouped according to the frequency of occurrence of each range, resulting in N numerical range groups, including:

[0021] Determine the range of the first value of N-1 groups, where the range of the first value of N-1 groups is the first N-1 value range in the first sequence, and the first sequence is obtained by sorting the different value ranges in descending order of the frequency of occurrence.

[0022] To determine the first N-1 value range groups from N value range groups, the steps for determining each value range group in the first N-1 value range groups include: starting from the first value range of the i-th group in the second sequence, searching in reverse order of the second sequence; if other first value ranges of other groups are found, then the first value range of the i-th group and the value range between the first value range of the i-th group and the other first value ranges of other groups are taken as the i-th value range group; otherwise, the first value range of the i-th group, the first value range in the second sequence, and the value range between the first value range of the i-th group and the first value range in the second sequence are taken as the i-th value range group; wherein, the second sequence is obtained by sorting the different value ranges in ascending order according to the difference they represent; the initial value of i is 1, and i is an integer less than or equal to N-1;

[0023] Determine the Nth value range group among N value range groups, where the Nth value range group includes value ranges that do not belong to the first N-1 value range groups.

[0024] Based on the above scheme, by searching for groups based on the frequency of occurrence from largest to smallest, we can find the maximum frequency of occurrence, thereby finding the array range with the most occurrences, and thus assigning it a code value with fewer bits, and also enabling this code value to represent a smaller number of bits.

[0025] In one possible implementation, before determining the frequency of occurrence of different numerical ranges based on multiple differences, the method further includes:

[0026] By selecting different values ​​of N, the steps of determining the encoding value corresponding to the difference based on the frequency of occurrence of the numerical range in which the difference falls are executed; and constructing the target data sequence based on the header data, the difference, and the encoding value corresponding to the difference are performed to obtain the target data sequence corresponding to different values ​​of N.

[0027] Based on the target data sequences corresponding to different values ​​of N, the compression ratio corresponding to each target data sequence is determined; whereby the compression ratio characterizes the rate of decrease in storage space occupied by the target data sequence relative to the data sequence to be processed;

[0028] The value of N corresponding to the highest compression ratio for each target data sequence is taken as the target value of N.

[0029] Based on the above scheme, and through multiple trials, the number of numerical ranges with the highest compression ratio was found to ensure the final compression ratio.

[0030] In one possible implementation, the target data sequence further includes: decoded information;

[0031] The decoding information includes the number of bits corresponding to each value range group in descending order of the total number of occurrences.

[0032] Based on the above scheme, by further combining the decoding information, the header data, the difference, and the corresponding encoded value of the difference, lossless compression can be achieved, and the original data can be restored without loss after subsequent storage or transmission.

[0033] In one possible implementation, acquiring the data sequence to be processed includes:

[0034] The image to be processed is acquired and parsed to obtain the data values ​​of each pixel in the image.

[0035] Arrange the data values ​​in either the first or second order to obtain the data sequence to be processed;

[0036] The first sequence represents extracting multiple pixel rows from the image to be processed sequentially in a first direction, and extracting the data values ​​of multiple pixels in each pixel row sequentially in a second direction; the second sequence represents extracting multiple pixel columns from the image to be processed sequentially in a second direction, and extracting the data values ​​of multiple pixels in each pixel column sequentially in a first direction; the first direction is from top to bottom or from bottom to top; the second direction is from left to right or from right to left.

[0037] Based on the above scheme, by splitting the image to be processed into rows or columns, a sequence of data to be processed can be obtained. This allows for compression processing of the image, reducing the storage space occupied by the image to be processed.

[0038] In one possible implementation, before arranging the data values ​​according to a first or second order to obtain the data sequence to be processed, the method further includes:

[0039] For the image to be processed, a first rate of change and a second rate of change are determined; wherein, the first rate of change is the gray-level gradient value of the pixel row in the image to be processed; and the second rate of change is the gray-level gradient value of the pixel column in the image to be processed.

[0040] If the first rate of change is determined to be less than or equal to the second rate of change, the data values ​​are arranged in the first order to obtain the data sequence to be processed; otherwise, the data values ​​are arranged in the second order to obtain the data sequence to be processed.

[0041] Based on the above scheme, by employing horizontal or vertical difference calculation strategies, the data sequence to be processed can be extracted in a way that achieves a better compression rate, according to the characteristics of the image to be processed. Dynamically selecting the optimal data extraction method reduces the difference range, further improving the data compression rate and thus reducing storage space usage.

[0042] In one possible implementation, the number of images to be processed is multiple. After arranging the data values ​​according to a first order or a second order to obtain a sequence of data to be processed, the method further includes:

[0043] The data values ​​of each pixel in one of the multiple images to be processed are arranged in a first order to obtain a first data sequence to be processed, and arranged in a second order to obtain a second data sequence to be processed.

[0044] For the first and second data sequences to be processed, the following steps are performed respectively: determining the header data and difference sequence based on the data sequences to be processed; determining the encoding value corresponding to the difference based on the number of times the difference falls within the numerical range; and constructing the target data sequence based on the header data, the difference, and the encoding value corresponding to the difference, thereby obtaining the corresponding first and second target data sequences.

[0045] Calculate the compression ratios for the first target data sequence and the second target data sequence, respectively; where the compression ratio represents the rate of decrease in storage space occupied by the target data sequence relative to the data sequence to be processed;

[0046] The splitting order corresponding to the higher compression ratio between the two is used as the splitting order of other images to be processed among multiple images to be processed; wherein, the splitting order is either the first order or the second order.

[0047] Based on the above scheme, when processing multiple images in a batch, the splitting direction with a higher compression ratio can be determined in one execution, ensuring a high compression ratio in the processing of multiple image data.

[0048] In a second aspect, this application provides a data processing apparatus configured to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0049] In one possible implementation, the data processing apparatus includes:

[0050] The acquisition module is used to acquire the data sequence to be processed; wherein, the data sequence to be processed includes multiple data to be processed;

[0051] The processing module is used to determine the header data and the difference sequence based on the data sequence to be processed; wherein, the header data is the first data to be processed in the data sequence to be processed; the difference sequence includes multiple differences, each of which is the difference between two adjacent data to be processed in the multiple data to be processed;

[0052] The processing module is also used to determine the encoding value corresponding to the difference based on the number of times the difference falls into the numerical range; wherein, the number of occurrences is the number of differences that fall into each numerical range among multiple differences; the encoding value is used to indicate the number of bits stored in the difference, and the number of bits in the encoding value is negatively correlated with the number of occurrences;

[0053] The processing module is also used to construct the target data sequence based on the header data, the difference, and the encoded value corresponding to the difference.

[0054] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0055] The memory stores the instructions that the computer executes;

[0056] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0057] Fourthly, this application provides a data structure, including: header data, difference, and encoded value corresponding to the difference;

[0058] The header data is the first data to be processed in the data sequence; the difference is the difference between two adjacent data to be processed in the data sequence; the encoding value corresponding to the difference is used to indicate the number of bits stored in the difference, and the number of bits of the encoding value is negatively correlated with the number of occurrences of the numerical range to which the difference belongs.

[0059] In one possible implementation, the data structure further includes: decoded information;

[0060] The decoding information includes the number of bits corresponding to the total number of occurrences of the numerical range group in descending order; the numerical range group includes at least one numerical range, which is the numerical range into which the difference value falls.

[0061] Fifthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0062] In a sixth aspect, this application provides a program product including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0063] The data processing method, data processing device, storage medium, and program product provided in this application embodiment acquire a data sequence to be processed, determine header data as the sequence start information and a difference sequence representing the change relationship between adjacent data to be processed, determine the encoding value corresponding to the difference based on the frequency of occurrence of the numerical range in which the difference falls, and then construct a target data sequence based on the header data, the difference, and the encoding value corresponding to the difference. This can utilize the small continuous fluctuation characteristics between adjacent data in the data sequence to be processed, and adopt a bit representation method that matches the frequency of occurrence of the difference with different distribution characteristics, thereby reducing data storage redundancy, improving lossless compression efficiency, and reducing storage resource occupation and transmission overhead. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 Flowchart of the data processing method provided in this application Figure 1 ;

[0066] Figure 2 Flowchart of the data processing method provided in this application Figure 2 ;

[0067] Figure 3 Flowchart of the data processing method provided in this application Figure 3 ;

[0068] Figure 4 Flowchart of the data processing method provided in this application Figure 4 ;

[0069] Figure 5 Flowchart of the data processing method provided in this application Figure 5 ;

[0070] Figure 6This is an exemplary first sequence diagram;

[0071] Figure 7 This is an exemplary second sequence diagram;

[0072] Figure 8 Flowchart of the data processing method provided in this application Figure 6 ;

[0073] Figure 9 A schematic diagram of the data processing apparatus provided in this application;

[0074] Figure 10 A schematic diagram of the structure of the electronic device provided in this application.

[0075] Explanation of reference numerals in the attached figures:

[0076] 90 - Data processing device;

[0077] 901 - Acquisition Module; 902 - Processing Module;

[0078] 100 - Electronic devices;

[0079] 1001 - Processor; 1002 - Memory; 1003 - Communication components; 1004 - Bus. Detailed Implementation

[0080] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0081] During data acquisition and processing, it is necessary to convert the analog data collected by relevant devices into digital data for storage. In the data acquisition process of various application fields, analog-to-digital converters (ADCs) can convert continuous analog signals into digital data.

[0082] Taking image data acquisition as an example, in the fields of optical imaging, biomedical detection, and semiconductor detection, image acquisition devices typically continuously output large amounts of pixel data and need to maintain the integrity of the original data in subsequent storage, transmission, or analysis stages. Related systems generally include an image acquisition unit, a data processing unit, and a storage or transmission unit. The acquired image data is converted into a data sequence arranged in a certain order before being encoded.

[0083] This application notes that during the aforementioned image data processing, the pixel values ​​of these raw image data typically exhibit small spatial fluctuations. These fluctuations may be caused by minute deformations of biological tissues, random disturbances in optical noise, or localized changes in semiconductor defects. Based on this characteristic, a data processing algorithm is provided. It should be understood that this application is not only applicable to the aforementioned image acquisition devices, but also applicable to any data exhibiting small fluctuations.

[0084] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0085] Figure 1 Flowchart of the data processing method provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0086] Step 101. Obtain the data sequence to be processed.

[0087] The data sequence to be processed includes multiple data items to be processed.

[0088] For example, the data sequence to be processed is used as the base data sequence for subsequent difference calculation and encoding processing, and the multiple data to be processed are data units arranged in a predetermined order.

[0089] The execution subject of the method provided in this application embodiment can be a processor, industrial control computer, server, or dedicated encoding chip.

[0090] For example, in a specific implementation, the data sequence to be processed can be the original data sequence obtained from an external device, or it can be the sequence obtained after sequential processing of the original data.

[0091] Specifically, a contiguous storage area can be allocated in the buffer, and the acquired data to be processed can be written sequentially in the order of reading to form a sequence of data to be processed containing multiple data items. To ensure consistency in subsequent difference calculations, a uniform data type is used to represent each data item within the same sequence.

[0092] In one possible embodiment, the data sequence to be processed may consist of a complete set of data; in another possible embodiment, the data sequence to be processed may also consist of a portion of continuous data.

[0093] In another possible embodiment, image data acquired by an optical device is used as an example. An image acquisition device acquires an image and performs analog-to-digital conversion to obtain a sequence of data to be processed. This sequence includes multiple data points to be processed. Taking image data as an example, the data to be processed can be the data values ​​corresponding to pixels in the image.

[0094] For example, each piece of data to be processed occupies 2 bytes of storage space, used to represent the data value of the corresponding pixel. This could be the grayscale value or color value of the pixel.

[0095] Optionally, each piece of data to be processed in the data sequence can be denoted as: b1, b2, b3, ..., bn-1, bn. It can be understood that there are n pieces of data to be processed in the data sequence. Each piece of data occupies 2 bytes of storage space. It should be understood that the above example is merely illustrative and not a limitation.

[0096] Based on the above analysis, it can be seen that by arranging the data to be processed in a sequence, the original data to be processed can participate in the analysis of differences between adjacent data in a sequential form.

[0097] Step 102. Determine the header data and difference sequence based on the data sequence to be processed.

[0098] The header data is the first data to be processed in the data sequence; the difference sequence includes multiple differences, each of which is the difference between two adjacent data to be processed in the multiple data to be processed.

[0099] For example, the header data is used as the starting reference value for restoring the difference sequence, which characterizes the relationship of change between adjacent data to be processed. In a specific implementation, after the data sequence to be processed is written, the first data to be processed in the sequence is read and directly recorded as the header data. It can be understood that the header data retains the original value in the target data sequence and is used to recover the first data to be processed during decoding.

[0100] Subsequently, following the order of the data sequence to be processed, adjacent difference calculations are performed on each data point starting from the second data point. Let the data sequence be X0, X1, X2…Xn, then the first data point is X0, and the i-th difference Di in the difference sequence can be obtained by subtracting Xi-1 from Xi, where i ranges from 1 to n. Through this calculation method, the original sequence is transformed into a combination of a baseline value and multiple variables. When the variation between adjacent data points is small, the multiple differences will be concentrated in a small numerical range, providing an input basis for subsequent allocation of encoded values ​​based on distribution characteristics.

[0101] It should be noted that the differences need to be able to represent positive, zero, and negative values; therefore, each difference in the difference sequence is stored in a signed data format. Header data extraction and difference sequence generation are completed simultaneously in a single traversal. That is, the first piece of data to be processed is read and written to the header data buffer, then subsequent pieces of data are read one by one, subtracted from the previous piece of data, and the results are written to the difference buffer. If the length of the data sequence to be processed is N, then the length of the difference sequence is N minus 1.

[0102] In one possible embodiment, the difference between two adjacent data to be processed is calculated by subtracting the preceding data from the following data; in another possible embodiment, the order of data can be agreed upon at the encoding and decoding ends.

[0103] In another possible embodiment, the first piece of data to be processed in the sequence of data to be processed is used as the header data. This can be understood as using b1 from the aforementioned data to be processed as the header data. Furthermore, based on each piece of data to be processed, the difference between adjacent pairs of data to be processed is calculated to obtain multiple difference values, forming a difference sequence.

[0104] Optionally, based on the above exemplary representation of the data sequence to be processed, subtracting b1 from the data to be processed b2 yields the difference c2,1, and subtracting b2 from the data to be processed b3 yields the difference c3,2. This process continues, resulting in a total of n-1 differences, thus forming a difference sequence. For example, the difference sequence can be denoted as: c2,1, c3,2, c4,3, ..., cn,n-1. It should be understood that the above examples are merely illustrative and not limiting.

[0105] Based on the above analysis, it can be seen that by retaining the header data and converting the remaining data into adjacent differences, the absolute representation of the original values ​​can be transformed into a representation of changes without compromising reproducibility. Subsequently, the original sequence can be recovered simply by using the header data and the sum of the differences.

[0106] Step 103. Determine the code value corresponding to the difference based on the frequency of occurrence of the numerical range into which the difference falls.

[0107] The occurrence count is the number of differences that fall within each numerical range; the encoding value is used to indicate the number of bits stored in the difference, and the number of bits in the encoding value is negatively correlated with the occurrence count.

[0108] For example, the numerical range is used to classify the differences, the frequency of occurrence is used to characterize the number of differences contained in each numerical range, and the encoded value is used to indicate the number of bits of storage for the corresponding difference in the target data sequence.

[0109] Below is an example of the occurrence count and encoded value.

[0110] For example, the data sequence to be processed is 0, 0, 2, 5, 10, 16, 23. The header data is 0, and there are 5 differences: 0, 2, 3, 5, 6, and 7. There are 3 sets of numerical ranges: range A: [0,1], range B: [2,3], and range C: [4,7].

[0111] In the above scenario, the frequency of occurrence of range A is 1 (0 falls within range A, therefore the number of differences within range A is 1), the frequency of occurrence of range B is 2 (2 and 3 fall within range B, therefore the number of differences within range B is 2), and the frequency of occurrence of range C is 3 (5, 6, and 7 fall within range C, therefore the number of differences within range C is 3). The encoding value of range C, which has the highest total frequency in the range group, can be set to one bit, for example, 0, and the storage bits for the difference indicated by the encoding value 0 can be defined as 3 bits (it should be understood that storing any number in [4,7] in binary generally requires 3 bits). The encoding value of range B, which has the second highest total frequency in the range group, can be set to two bits, for example, 10, and the storage bits for the difference indicated by the encoding value 10 can be defined as 2 bits (it should be understood that storing any number in [2,3] in binary generally requires 2 bits). The encoding value of the third most frequent value range A in the total number of occurrences can be set to three bits, such as 110, and the storage bits for the difference indicated by the encoding value 110 can be defined as 1 bit (it should be understood that storing any number in [0,1] in binary generally requires 1 bit).

[0112] For example, in practical implementation, the difference sequence is first divided into intervals. The numerical range can be divided according to predetermined rules so that each difference falls within the corresponding numerical range. After traversing all differences in the difference sequence, the number of differences in each numerical range is counted to obtain the corresponding occurrence frequency.

[0113] Subsequently, an encoding mapping relationship is established according to the frequency of occurrence of each numerical range, so that numerical ranges with higher frequency of occurrence correspond to encoded values ​​with fewer bits, and numerical ranges with lower frequency of occurrence correspond to encoded values ​​with more bits. The encoded value itself can be a prefix identifier, a category identifier, or a bit width indicator field, and its core function is to indicate how many bits to use for storing the corresponding difference.

[0114] In one possible implementation, a statistical table is first generated, recording the interval boundaries, frequency of occurrence, and assigned coded value for each numerical range. Then, the multiple numerical ranges are sorted according to their frequency of occurrence, with the most frequent ranges being assigned shorter coded values. For example, the numerical range with the highest frequency of occurrence can correspond to a coded value with fewer bits, while the second most frequent range can correspond to a coded value with more bits, thus satisfying the constraint that the number of bits in the coded value is negatively correlated with the frequency of occurrence.

[0115] Negative correlation here refers to the fact that in the encoding configuration of the same target data sequence, when the first numerical range appears more frequently than the second numerical range, the number of bits of the encoded value assigned to the first numerical range is no greater than the number of bits of the encoded value assigned to the second numerical range.

[0116] Furthermore, once the encoded value is determined, the corresponding encoded value is written for each difference according to the numerical range in which each difference exists, and the difference is stored numerically according to the number of bits indicated by the encoded value.

[0117] In some cases, if the number of significant bits of the difference is less than the number of bits of storage indicated by the encoded value, it is expanded or padded in a conventional manner; if the difference falls into a larger value range, it is stored using a longer encoded value associated with that value range and the corresponding bit width.

[0118] In another possible embodiment, since the difference values ​​differ, larger differences require more storage bits during storage. For example, a difference within the range [0,1] requires 1 bit of storage space, while a difference within the range [-2,1] requires 2 bits of storage space.

[0119] Optionally, the number of numerical ranges to be divided can be determined based on the maximum value of the difference sequence. For example, if the maximum value of the difference sequence requires 16 bits of storage space, then the corresponding numerical range is [-32768, 32767].

[0120] For example, in a difference sequence, there are n-1 differences, of which 20 differences fall within the range of [0,1]. Therefore, the number of differences that occur in the range [0,1] is 20.

[0121] A corresponding code value can be assigned to a numerical range based on the frequency of its occurrences. It should be noted that, for the code value, the number of bits in the code value corresponding to that numerical range is negatively correlated with the frequency of its occurrences.

[0122] It's understandable that if the differences within a numerical range occur frequently, a smaller number of bits are needed for encoding, thus compressing the storage space required. Furthermore, the encoded value corresponding to the numerical range is assigned to the differences belonging to that range to obtain the encoded value for each difference in the difference sequence. The encoded value of the difference indicates the number of bits required for storage during the storage process. It should be understood that the above example is merely illustrative and not limiting.

[0123] Based on the above analysis, it can be seen that by constructing coding rules around the difference distribution, the frequently occurring difference ranges can be expressed with shorter identifiers and more matching bit widths, thereby compressing redundant high-bit information.

[0124] Step 104. Construct the target data sequence based on the header data, the difference, and the corresponding encoded value of the difference.

[0125] Taking the data sequence to be processed as 0, 0, 2, 5, 10, 16, 23 as an example, the differences are 0, 2, 3, 5, 6, 7. The target data sequence can be 0110010101011010101100111. Specifically, the corresponding interpretations are: 0 (header data) 110 (encoded value of difference 0) 0 (binary representation of difference 0) 10 (encoded value of difference 2) 10 (binary representation of difference 2) 10 (encoded value of difference 3) 11 (encoded value of difference 3) 0 (encoded value of difference 5) 101 (binary representation of difference 5) 0 (encoded value of difference 6) 110 (binary representation of difference 6) 0 (encoded value of difference 7) 111 (binary representation of difference 7). It should be understood that the data to be processed here is only a simple example, and its small fluctuation characteristics are not obvious. In practical applications, for data with more obvious small fluctuation characteristics, the method of this application is more effective.

[0126] For example, the target data sequence is used to store or transmit the processed result data and carries the information needed to subsequently restore the original data sequence to be processed. In a specific implementation, the header data is first written into the output buffer as the beginning of the target data sequence. The header data is stored with the original bit width to ensure that the first piece of data to be processed can be accurately restored.

[0127] Subsequently, following the original order of the differences in the difference sequence, the coded value corresponding to each difference and the difference data at the number of bits indicated by that coded value are written sequentially. In this way, a sequential structure of "header data, coded value, difference data" is formed within the target data sequence, in which the coded value and difference data corresponding to each difference are matched one-to-one and maintain consistency with the positional relationship in the data sequence to be processed.

[0128] Correspondingly, at the receiving or decoding end, the header data is first read as the initial recovery value, and then the subsequent encoded values ​​are parsed in sequence. The number of bits to be read for the current difference data is determined according to the encoded value. After taking out the difference, it is added to the previous recovery data to obtain the current recovery data. After processing item by item, the original data sequence to be processed is recovered.

[0129] In one possible embodiment, during specific encapsulation, the target data sequence can be stored in either a continuous bitstream format or a byte-aligned format. When using a continuous bitstream format, the encoded value and difference data are concatenated bit-by-bit and written to the output buffer; the buffer is output sequentially after each byte is filled. When using a byte-aligned format, padding can be added to predetermined boundaries after each set of encoded values ​​and difference data to adapt to the access rules of a specific bus or storage medium.

[0130] In some cases, in addition to header data, differences, and their corresponding encoded values, the target data sequence may also contain necessary configuration information for decoding; when the relevant rules are agreed upon in advance at the encoding and decoding ends, the target data sequence only needs to store header data, each difference, and its corresponding encoded value.

[0131] In another possible embodiment, the header data, each difference in the difference sequence, and the corresponding encoded value for each difference are used to construct a target data sequence corresponding to the data sequence to be processed. The target data sequence occupies less storage space than the data sequence to be processed. This is because it is not necessary to allocate a fixed number of bits of storage for each piece of data to be processed; instead, the encoded value indicates the number of bits of storage required for each difference. This avoids redundancy in storage space usage.

[0132] Furthermore, the target data sequence is stored in a preset storage device. Optionally, the target data sequence in the storage device is transmitted to the target device. Since the target data sequence is compressed compared to the data sequence to be processed, the efficiency of data transmission can also be improved. It should be understood that the above example is only illustrative and not limiting.

[0133] Based on the above analysis, it can be seen that by combining the header data, the difference, and the corresponding encoded value of the difference in an orderly manner, the target data sequence not only retains the complete information required for the original data to be restored, but also makes the storage bits of each difference match its distribution range, thereby reducing storage and transmission costs under the constraint of lossless restoration.

[0134] The data processing method provided in this application, in some cases (especially when the data sequence to be processed consists of small fluctuations with little difference between consecutive data), can transform the original data to be processed into a difference sequence with smaller values, thus requiring fewer bits for storage and being easier to compress, by performing subtraction. Furthermore, by retaining the header data to ensure the integrity of the difference sequence, a baseline value is provided for subsequent decompression. Finally, the encoded value indicates the number of bits required to store the difference, and a target data sequence is constructed based on the header data, the difference, and the encoded value. This facilitates understanding how to parse the data when parsing the target data sequence. Moreover, since the encoded value of the difference is negatively correlated with its frequency of occurrence, frequently occurring differences can be encoded with fewer bits, improving the compression ratio, thereby reducing the space occupied by data storage and improving data transmission efficiency.

[0135] For images such as optical grayscale images, where the data is characterized by small fluctuations, and adjacent data exhibit small, continuous fluctuations, a large number of differences are concentrated within a few small numerical ranges. Consequently, the corresponding encoded values ​​and storage bits are shortened. Meanwhile, a small number of significantly changing differences are preserved and fully represented using longer encoded values. Therefore, while ensuring that the original data can be restored item by item based on the header data and differences, the high-bit redundancy in the fixed-length recording method is reduced, the data compression rate is improved, and thus storage and transmission efficiency is enhanced.

[0136] Figure 2 Flowchart of the data processing method provided in this application Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the illustrated embodiment, the process of determining the encoded value corresponding to the difference in step 103 above will be described in detail. The method includes:

[0137] Step 201. Determine the frequency of occurrence of different numerical ranges based on multiple differences.

[0138] Step 202. Based on the frequency of occurrence of each numerical range, group the different numerical ranges to obtain N numerical range groups.

[0139] Each numerical range group includes at least one numerical range, and N is an integer greater than 1.

[0140] Step 203. For the difference that falls within the numerical range, set the code value corresponding to the numerical range group to which the numerical range belongs, so as to obtain the code value corresponding to the difference.

[0141] For example, in specific implementation, the range of values ​​for each difference in the difference sequence can be determined first, and it can be classified into a preset numerical range. Then, the number of differences in each numerical range can be counted to obtain the distribution of occurrences. The numerical ranges can be sorted according to the number of occurrences, and numerical ranges with similar occurrences or the same number of bits in their encoded values ​​can be grouped into the same numerical range group. Then, a corresponding encoded value can be assigned to each numerical range group.

[0142] Furthermore, the encoded value can use a fixed-length group identifier or a variable-length group identifier. Each difference within a group shares this group identifier during encoding, thus forming the encoded value corresponding to the difference. For numerical ranges that occur frequently, a shorter group identifier can be assigned; for numerical ranges that occur infrequently, a longer group identifier can be assigned. The specific number of bits is determined based on the statistical results of the difference distribution.

[0143] It can be understood that when constructing the target data sequence, based on the correspondence between numerical range groups and encoded values, each difference is replaced with the encoded value of its corresponding numerical range group, and written into the target data sequence along with the header data. During decoding, the initial data is recovered based on the header data, the corresponding numerical range group is identified based on the encoded value, and the difference is restored by combining the range mapping relationship within the numerical range group, ultimately reconstructing the original data sequence.

[0144] In one possible embodiment, the number of differences across different numerical ranges is counted as the frequency of occurrence of each numerical range. Based on the foregoing exemplary description, the maximum value of the difference requires 16 bits of storage space; following the aforementioned pattern, the required storage space for different numerical ranges can be obtained.

[0145] Specifically, the required storage space for different numerical ranges is as follows:

[0146] If the difference is within the range [0,1], it requires 1 bit of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n1.

[0147] Aside from the range already mentioned (i.e., the [0,1] interval), when the difference falls within the [-2,1] interval, its storage requires 2 bits of space. Optionally, the number of differences (occurrences) within this range can be denoted as n². It should be understood that the storage space required for the range already mentioned (i.e., the [0,1] interval) is already determined (here, 1 bit), therefore, this discussion focuses on how to determine the storage space for value ranges outside the aforementioned range.

[0148] Aside from the ranges already mentioned, if the difference falls within the range [-4, 3], its storage requires 3 bits of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n3.

[0149] Aside from the ranges already mentioned, if the difference falls within the range [-8, 7], it requires 4 bits of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n4.

[0150] Aside from the ranges already mentioned, if the difference falls within the range of [-16, 15], it requires 5 bits of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n5.

[0151] Aside from the ranges already mentioned, if the difference falls within the range [-32, 31], its storage requires 6 bits of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n6.

[0152] Aside from the ranges already mentioned, if the difference falls within the range of [-64, 63], it requires 7 bits of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n7.

[0153] Aside from the ranges already mentioned, if the difference falls within the range of [-128, 127], its storage requires 8 bits of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n8.

[0154] Aside from the ranges already mentioned, if the difference falls within the range of [-256, 255], it requires 9 bits of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n9.

[0155] Aside from the ranges already mentioned, if the difference falls within the range of [-512, 511], its storage requires 10 bits of storage space. Optionally, the number of differences (occurrences) falling within this range can be denoted as n10.

[0156] Aside from the ranges already mentioned, if the difference falls within the range of [-1024, 1023], it requires 11 bits of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n11.

[0157] Aside from the ranges already mentioned, if the difference falls within the range of [-2048, 2047], it requires 12 bits of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n12.

[0158] Aside from the ranges already mentioned, if the difference falls within the range of [-4096, 4095], it requires 13 bits of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n13.

[0159] Aside from the ranges already mentioned, if the difference falls within the range of [-8192, 8191], it requires 14 bits of storage space. Optionally, the number of differences (occurrences) within this range can be denoted as n14.

[0160] Aside from the ranges already mentioned, if the difference falls within the range of [-16384, 16383], its storage requires 15 bits of storage space. Optionally, the number of differences (occurrences) falling within this range can be denoted as n15.

[0161] Aside from the ranges already mentioned, if the difference falls within the range of [-32768, 32767], its storage requires 16 bits of storage space. Optionally, the number of differences (occurrences) falling within this range can be denoted as n16.

[0162] Arrange the above numerical ranges in descending order of frequency to obtain a sequence of numerical range groups. Divide the numerical ranges into multiple numerical range groups. Each numerical range group has a corresponding encoded value. The encoded value indicates the number of storage bits required for the numerical range with the largest number of storage bits within that group. Each numerical range group includes at least one numerical range.

[0163] The number of numerical range groups obtained is N, and the value of N can be set to an integer greater than 1. Optionally, N can be set to 4, that is, the first numerical range group includes 4 numerical ranges with the first to fourth frequency of occurrence, the second numerical range group includes 4 numerical ranges with the fifth to eighth frequency of occurrence, the third numerical range group includes 4 numerical ranges with the ninth to twelfth frequency of occurrence, and the fourth numerical range group includes 4 numerical ranges with the thirteenth to sixteenth frequency of occurrence.

[0164] Each of the aforementioned numerical range groups has a corresponding coded value, and the number of bits in the coded values ​​for the first to fourth numerical range groups gradually increases. Furthermore, for the numerical range to which the difference falls, the magnitude of the coded value is determined.

[0165] For example, the difference between the four value ranges falling into the first value range group is assigned the code value for the first value range group. The difference between the four value ranges falling into the second value range group is assigned the code value for the second value range group. And so on, so that the code value for each difference can be obtained.

[0166] Optionally, different grouping methods and different numbers of groups can be tried for multiple numerical ranges. For example, N can be set to an even number greater than 1. By dividing the numerical range into an even number of groups, such as 4 groups, a better compression rate can be achieved for 16-bit data.

[0167] In the above embodiments, by statistically analyzing the frequency of differences and grouping numerical ranges accordingly, the encoded values ​​can match the distribution characteristics of the differences. Differences with a higher frequency of occurrence within a numerical range can be represented by encoded values ​​with fewer bits, while differences with a lower frequency of occurrence are represented by encoded values ​​with more bits corresponding to the numerical range group. This reduces encoding redundancy and maintains data reversibility. By grouping numerical ranges and assigning the encoded value corresponding to the numerical range group to the difference, redundant storage space occupation caused by multiple encodings of the numerical range is avoided.

[0168] In the above Figure 2 Based on the illustrated embodiment, prior to step 201 above, the method may further include:

[0169] Determine the total number of occurrences for each value range group. The total number of occurrences is the sum of the number of occurrences of each value range included in the value range group.

[0170] The encoding value corresponding to a value range group is determined based on the total number of occurrences of that group. The encoding value indicates the maximum number of bits required for storage within that value range group; the number of bits in the encoding value increases as the total number of occurrences of the value range group decreases.

[0171] For example, in specific implementation, the frequency of occurrence of different numerical ranges can be counted first, and then numerical ranges with similar frequency of occurrence and the same or similar storage bit requirements can be grouped into the same numerical range group. The frequency of occurrence of each numerical range in the group can be summed to obtain the total frequency of occurrence of the group.

[0172] Subsequently, an encoding mapping relationship can be established based on the total occurrence frequency of each numerical range group. The numerical range group with a larger total occurrence frequency is assigned a shorter encoding value, and the numerical range group with a smaller total occurrence frequency is assigned a longer encoding value, so that the number of bits in the encoding value is inversely related to the total occurrence frequency of the numerical range group.

[0173] For example, the encoded value can be represented by binary code, or by other codewords that can uniquely identify a range of values. The specific codeword length can be pre-configured by the encoding table or dynamically generated by statistical results. In practical applications, the specific generation method of the encoded value can also be selected from other encoding rules, which are not limited in this application.

[0174] In one possible implementation, the occurrence of each numerical range is first obtained based on the difference distribution. Then, the ranges in the same group are summarized and calculated to generate a group-level encoded value. Subsequently, this encoded value and the corresponding difference are used together to construct the target data sequence. Since the encoded value can directly indicate the maximum number of bits required within the group, the number of bits required can be quickly determined and the data encapsulation completed according to the group-level encoding when storing or transmitting the difference.

[0175] In another possible embodiment, taking the division into 4 numerical range groups as an example, the first numerical range group includes 4 numerical ranges with the highest frequency of occurrence from the first to the fourth, and the total frequency of occurrence of the first numerical range group is the sum of the frequency of occurrence of the 4 numerical ranges with the highest frequency of occurrence from the first to the fourth.

[0176] The second numerical range group includes four numerical ranges that rank fifth to eighth in terms of frequency of occurrence. The total number of occurrences in the second numerical range group is the sum of the number of occurrences of the four numerical ranges that rank fifth to eighth in terms of frequency of occurrence.

[0177] The third numerical range group includes four numerical ranges that rank ninth to twelfth in terms of frequency of occurrence. The total number of occurrences in the third numerical range group is the sum of the number of occurrences in the four numerical ranges that rank ninth to twelfth in terms of frequency of occurrence.

[0178] The fourth numerical range group includes four numerical ranges that rank from thirteenth to sixteenth in terms of frequency of occurrence. The total number of occurrences in the fourth numerical range group is the sum of the number of occurrences in the four numerical ranges that rank from thirteenth to sixteenth.

[0179] Set the code value of the numerical range group with the highest total frequency to one digit, such as 0. Set the code value of the numerical range group with the second highest total frequency to two digits, such as 10. Set the code value of the numerical range group with the third highest total frequency to three digits, such as 110. Set the code value of the numerical range group with the fourth highest total frequency to four digits, such as 1110.

[0180] Optionally, since the number of differences in the fourth most frequent value range is small, its corresponding code value can also be set to three bits, such as 111.

[0181] The encoded value corresponding to each numerical range group indicates the maximum number of bits required for storage in that numerical range group.

[0182] For example, in the numerical range group that ranks first in total occurrences, there may be numerical ranges that require 1 bit, 2 bits, 4 bits, and 6 bits of storage respectively. In this case, the encoded value 0 of the numerical range group is used to indicate that the difference needs to be parsed using 6 bits.

[0183] Optionally, computation can be stored using Huffman coding: for example, a coding value of 0 is used to parse 16-bit data, a coding value of 10 is used to parse 12-bit data, a coding value of 110 is used to parse 8-bit data, and a coding value of 111 is used to parse 4-bit data.

[0184] In the example above, a grouping strategy based on the frequency of occurrence is used to group numerical ranges, and dynamic encoding optimization is performed based on these range groups. This allows differences that occur more frequently within a numerical range to be assigned shorter encoding values. The encoding allocation of the numerical range groups matches the actual frequency of occurrence, adapting encoding resource allocation to the difference distribution and reducing redundant bits in the target data sequence. This also enables data compression, reducing storage redundancy.

[0185] In some cases, the following situations may occur during the grouping process. For example, the range of values ​​n4 appears most frequently among all ranges, so it can be considered as a range group, let's call it group1. The range of values ​​n3 appears the second least frequently, but the range group containing the range of values ​​n3 (let's call it group2) should also include the ranges of values ​​n2 and n1.

[0186] If the total occurrence count of group2 is higher than that of group1, then the number of bits in the encoded value of group2 should be less than that of group1. It can be understood that in the process of grouping numerical ranges, it is necessary not only to consider the number of times the difference falls within a numerical range, but also to consider which other numerical ranges should be included in each numerical range group.

[0187] Figure 3 Flowchart of the data processing method provided in this application Figure 3 ,like Figure 3 As shown, based on the above embodiments and examples, step 202, which involves grouping N numerical range groups, can specifically include:

[0188] Step 301. Determine the range of the first values ​​of N-1 groups.

[0189] Among them, the first value range of N-1 groups is the first N-1 value range in the first sequence, and the first sequence is obtained by sorting the different value ranges in descending order of the frequency of occurrence.

[0190] Step 302. Determine the first N-1 value range groups among the N value range groups.

[0191] The steps for determining each of the first N-1 numerical range groups include:

[0192] Starting from the first value range of the i-th group in the second sequence, search in reverse order of the second sequence. If other first value ranges of the group are found, the first value range of the i-th group and the value range between the first value range of the i-th group and other first value ranges of the group are taken as the i-th value range group.

[0193] Otherwise, the range of the first value in the i-th group, the first value range in the second sequence, and the range of values ​​between the first value range of the i-th group and the first value range in the second sequence are taken as the i-th value range group.

[0194] The second sequence is obtained by sorting the different numerical ranges in ascending order of the differences they represent. The initial value of i is 1, and i is an integer less than or equal to N-1.

[0195] Step 303. Determine the Nth value range group among the N value range groups.

[0196] The Nth numerical range group includes numerical ranges that do not belong to the first N-1 numerical range groups.

[0197] For example, arranging multiple numerical ranges in descending order of frequency yields a first sequence. Arranging multiple numerical ranges in ascending order of the differences they represent yields a second sequence.

[0198] First, based on the number N of numerical range groups to be divided, the first N-1 numerical ranges are selected from the first sequence as the group leader numerical ranges. These group leader numerical ranges will then be divided into N-1 numerical range groups.

[0199] For example, each range of values ​​is numbered. For instance, the range 1 has the number of differences (i.e., the frequency of occurrences) of its ranges as n1. This process is repeated to obtain five ranges with their respective numbers and frequencies. It should be noted that the number of ranges mentioned above is merely an example.

[0200] The order of the numerical ranges in the first sequence is: range4, range3, range5, range2, and range1; the order of the numerical ranges in the second sequence is: range1, range2, range3, range4, and range5. Now, to divide these five numerical ranges into three groups, we select range4 and range3 from the first sequence as the first numerical range of each group. It can be understood that range4 is the first numerical range of the first group, and range3 is the first numerical range of the second group.

[0201] Based on the first group's first value range range4, the search proceeds in reverse direction in the second sequence (i.e., along the direction from large to small, representing the difference between the values ​​represented by the value ranges in the second sequence). The other group's first value range range3 is immediately found. At this point, the value range between the first group's first value range range4 and the other group's first value range ranges, together with the first group's first value range range4, are taken as the first value range group.

[0202] It should be noted that in the above scenario, if there are no other numerical ranges between the first value range of the first group and the first value ranges of other groups in the second sequence, then the first value range group only contains the first value range of the first group. However, in some cases, there are other numerical ranges between the first value range of the first group and the first value ranges of other groups in the second sequence. In this case, the first value range group includes: the first value range of the first group, and the aforementioned other numerical ranges between the first value range of the first group and the first value ranges of other groups in the second sequence.

[0203] Furthermore, based on the second group's first value range range3, the search proceeds in reverse in the second sequence. No other group's first value range is found. At this point, the second group's first value range range3, the first value range range1 in the second sequence, and the other value ranges (4range2) between them are collectively regarded as the second value range group.

[0204] Therefore, the first N-1 value range groups have been determined out of the N value range groups, and the Nth value range group is further determined. The Nth value range group includes value ranges that do not belong to the first N-1 value range groups, namely the value range range5. Thus, N value ranges are determined.

[0205] In the above embodiments, by grouping based on the frequency of occurrence from largest to smallest, the maximum frequency can be found, thus identifying the range of values ​​with the most occurrences. This allows for the assignment of a code value with fewer bits, and the code value can represent a smaller number of bits. This grouping method determines the group head based on the frequency of occurrence, and then divides the group boundaries according to the difference size, enabling high-frequency value ranges to preferentially form independent or continuous groups, while the remaining value ranges are concentrated and merged into the Nth group. Therefore, when setting code values ​​for differences later, the encoding mapping can be directly performed based on the value range group to which it belongs, thereby matching the number of bits in the code value with the difference distribution and reducing redundant bits in the target data sequence.

[0206] Figure 4 Flowchart of the data processing method provided in this application Figure 4 ,like Figure 4 As shown, based on the foregoing embodiments or examples, before determining the frequency of occurrence of different numerical ranges according to multiple differences, the method further includes:

[0207] Step 401. Select different values ​​of N, and perform the steps of determining the encoding value corresponding to the difference based on the number of times the difference falls within the numerical range; and constructing the target data sequence based on the header data, the difference, and the encoding value corresponding to the difference, to obtain the target data sequence corresponding to different values ​​of N.

[0208] Step 402. Determine the compression ratio of each target data sequence based on the different values ​​of N.

[0209] Compression ratio represents the rate of decrease in storage space occupied by the target data sequence relative to the data sequence to be processed.

[0210] Step 403. Take the value of N that corresponds to the highest compression ratio for each target data sequence as the target value of N.

[0211] For example, the data sequence to be processed can be a pixel sequence output by an image acquisition device. First, group coding and target data sequence construction are performed on different candidate N values ​​respectively. Then, the number of bytes occupied by each target data sequence is counted and compared with the number of bytes of the data sequence to be processed to obtain the compression ratio.

[0212] The compression ratio can be calculated by subtracting the storage space of the target data sequence from the storage space of the original data sequence to be processed, and then dividing by the storage space of the original data sequence to be processed, or by calculating according to the equivalent definition of the descent rate.

[0213] It is understandable that after obtaining a set of target data sequences for each candidate N value, the N corresponding to the highest compression ratio is selected as the target value, and this target value is fixed for subsequent statistics on the frequency of occurrence of numerical ranges, determination of encoded values, and construction of target data sequences, so as to match the encoding granularity with the difference distribution.

[0214] For example, in specific implementation, candidate N values ​​can be pre-set as multiple discrete integers. The encoding module iteratively executes encoding rules under different N values ​​on the same difference sequence, the storage module records the length or number of bytes of each target data sequence, and the comparison module outputs the N value with the highest compression ratio. This method allows the number of groups to be screened before formal encoding, first calculating the encoding results with different N values, and then selecting the optimal parameters based on the compression ratio, thereby providing stable target parameters for subsequent lossless compression.

[0215] In the above embodiments, the target value of N that is more suitable for the current difference distribution can be determined before encoding. Based on multiple attempts, the number of numerical range groups with the highest compression rate can be found, so that the storage space reduction rate of the target data sequence reaches a high level, thereby improving the overall compression efficiency and providing a consistent encoding parameter basis for maintaining data reproducibility in the future.

[0216] Based on the foregoing embodiments or examples, a separate field can be configured in the constructed target data sequence to record the decoding bit information corresponding to each current value range group.

[0217] In one example, the target data sequence also includes decoded information.

[0218] The decoding information includes the number of bits corresponding to each value range group in descending order of the total number of occurrences.

[0219] For example, in specific implementation, after completing the total occurrence count of each value range group, the value range groups are sorted from largest to smallest according to the total occurrence count, and the sorted group order and corresponding decoding bit number are written into the decoding information one by one.

[0220] Since numerical ranges with a higher total frequency typically correspond to higher frequency difference intervals, their decoding bit depth can be configured to be shorter, while numerical ranges with a lower total frequency can be configured to have a longer decoding bit depth. During decoding, the bit depth of the encoded value corresponding to different numerical ranges can be identified and decoded based on the relationship recorded in the decoding information. The specific storage format of the decoded information can be a continuous bit string, a byte sequence, or a field table. Other formats can also be chosen in practical applications, and this application does not impose any limitations on this.

[0221] In one possible embodiment, taking the division of numerical ranges into four groups as an example, each numerical range group has a corresponding encoded value, and each encoded value has bit information about the difference that should be parsed. Therefore, in the decoded information, the parsed bit information corresponding to each encoded value can be set to a length of 4 bits. Consequently, the total length of the decoded information is 4 × 4 bits.

[0222] Specifically, one possible decoding information is represented as "16bit 12bit 8bit 4bit", which means that the encoded value corresponding to the first value range group should be decoded as a 16-bit difference, the encoded value corresponding to the second value range group should be decoded as a 12-bit difference, the encoded value corresponding to the third value range group should be decoded as an 8-bit difference, and the encoded value corresponding to the fourth value range group should be decoded as a 4-bit difference.

[0223] The decoded information, together with the header data, differences, and encoded values ​​in the target data sequence, constitutes a complete reversible representation. After reading the decoded information, the decoding end first restores the bit mapping relationship of each numerical range group, then combines the encoded values ​​corresponding to the differences to parse the difference sequence, thereby reconstructing the data sequence to be processed. Thus, the target data sequence retains the key information required for decoding while being compressed and stored, enabling subsequent lossless recovery.

[0224] In the example above, the target data sequence has built-in decoding information, allowing the decoding end to identify the corresponding number of bits for each numerical range group without additional negotiation of encoding rules. This ensures the reproducibility of the encoding result, enables lossless compression, and guarantees that the original data can be losslessly restored after subsequent storage or transmission. Furthermore, it allows numerical range groups with different frequencies to be parsed according to their corresponding number of bits, improving the decodeability and processing consistency of the target data sequence.

[0225] As can be seen from the foregoing explanation, the data processing method provided in this application embodiment can be applied to image processing. In such a scenario, Figure 5 Flowchart of the data processing method provided in this application Figure 5 ,like Figure 5As shown, obtaining the data sequence to be processed can specifically include:

[0226] Step 501. Obtain the image to be processed and perform parsing to obtain the data values ​​of each pixel in the image to be processed.

[0227] Step 502. Arrange the data values ​​in either the first or second order to obtain the data sequence to be processed.

[0228] The first sequence represents extracting multiple pixel rows from the image to be processed sequentially in a first direction, and extracting the data values ​​of multiple pixels in each pixel row sequentially in a second direction; the second sequence represents extracting multiple pixel columns from the image to be processed sequentially in a second direction, and extracting the data values ​​of multiple pixels in each pixel column sequentially in a first direction; the first direction is from top to bottom or from bottom to top; the second direction is from left to right or from right to left.

[0229] For example, in a specific implementation, after acquiring the image to be processed, the image parsing unit can convert the image file into a pixel matrix and read the data values ​​of each pixel in the matrix.

[0230] In some cases, when the first order is used, pixel rows are extracted one by one in the first direction, and then the pixel data values ​​in each row are read sequentially in the second direction, thus forming a continuous sequence of data to be processed.

[0231] In other cases, when the second order is used, the pixel columns are extracted one by one in the second direction, and then the pixel data values ​​in each column are read sequentially in the first direction, thus forming another data sequence to be processed.

[0232] The first and second directions can be determined based on the image storage format, acquisition scanning direction, or subsequent compression processing requirements. For example, when the first direction is set to top-to-bottom and the second direction is set to left-to-right, the sequence expands row by row. When the direction combination changes, the starting position and expansion path of the sequence change accordingly, but the complete extraction of the original pixel data is maintained. By converting the two-dimensional image into a one-dimensional data sequence, the relationship between adjacent data values ​​can be directly processed subsequently.

[0233] Combination Figure 6 and Figure 7 The first and second orders are explained separately. Figure 6 This is an exemplary first sequence diagram. Figure 7 This is an exemplary second-order schematic diagram. Figure 6 and Figure 7 In this diagram, each square represents a pixel in the image to be processed, and the number in each square indicates the extraction order of the data values ​​corresponding to the pixel.

[0234] like Figure 6 As shown in [a], pixel rows are extracted from top to bottom, and the data values ​​of the pixels are extracted from left to right for each pixel row.

[0235] like Figure 6 As shown in [b], pixel rows are extracted from bottom to top, and the data values ​​of the pixels are extracted from left to right for each pixel row.

[0236] like Figure 6 As shown in [c], pixel rows are extracted from top to bottom, and the data values ​​of the pixels are extracted from each pixel row in a right-to-left order.

[0237] like Figure 6 As shown in [d], pixel rows are extracted from bottom to top, and the data values ​​of the pixels are extracted from right to left for each pixel row.

[0238] like Figure 7 As shown in [a], the pixel columns are extracted from left to right, and the data values ​​of the pixels are extracted from top to bottom for each pixel column.

[0239] like Figure 7 As shown in [b], pixel columns are extracted from right to left, and the data values ​​of pixels are extracted from top to bottom for each pixel column.

[0240] like Figure 7 As shown in [c], the pixel columns are extracted from left to right, and the data values ​​of the pixels are extracted from bottom to top for each pixel column.

[0241] like Figure 7 As shown in [d], the pixel columns are extracted from right to left, and the data values ​​of the pixels are extracted from bottom to top for each pixel column.

[0242] In one possible embodiment, the image to be processed may be a black and white image or a color image, and the pixel values ​​are represented differently for different types of images. In one example, the image to be processed is a black and white image, and the data values ​​of the pixels are grayscale values; or the image to be processed is a color image, and the data values ​​of the pixels are color space data values.

[0243] In one possible implementation, the image to be processed is a black and white image, and the data value of each pixel in the image to be processed mentioned in the foregoing example is the grayscale value of that pixel.

[0244] In one possible implementation, the image to be processed is a color image, and the data of each pixel in the image to be processed mentioned in the foregoing example is the color space data value of that pixel.

[0245] Optionally, the color space can be the additive red-green-blue (RGB) color space or the subtractive CMYK color space. Color space data values ​​refer to the values ​​corresponding to different colors in the color space.

[0246] In the above example, the meaning of the pixel data value can be defined differently depending on the type of image, which can meet the requirements of compressed storage of digital conversion of images to be processed in various scenarios.

[0247] Optionally, as shown in the previous examples, for video or audio storage, analog-to-digital conversion can also be performed to obtain the data sequence to be processed. Specifically, taking video storage as an example, the video to be processed is acquired and parsed to obtain video frames at different times in the video to be processed, and the video frames at different times are digitized to obtain the data sequence to be processed.

[0248] Specifically, taking audio storage as an example, the audio to be processed is obtained and parsed to obtain the audio amplitude values ​​at different times in the audio to be processed, thus obtaining the data sequence to be processed.

[0249] In the above embodiments, the pixel data in the image to be processed can form a data sequence to be processed in a determined scanning order. Based on the pixel data of the image to be processed, either row-wise or column-wise, the data sequence to be processed can be obtained. This allows for image compression, reducing the storage space occupied by the image to be processed. It also facilitates adaptation to different image features based on different orientations of the arrangement.

[0250] As can be seen from the above embodiments, after extracting the data values ​​from the image to be processed, before arranging them, it is also possible to choose whether to sort in the first order or the second order based on the image change characteristics of the image to be processed.

[0251] Specifically, in one example, before arranging the data values ​​in a first or second order to obtain the sequence of data to be processed, the method further includes:

[0252] For the image to be processed, determine a first rate of change and a second rate of change. If the first rate of change is less than or equal to the second rate of change, arrange the data values ​​in a first order to obtain the data sequence to be processed; otherwise, arrange the data values ​​in a second order to obtain the data sequence to be processed.

[0253] Wherein, the first rate of change is the gray-level gradient value of the pixel row in the image to be processed; the second rate of change is the gray-level gradient value of the pixel column in the image to be processed;

[0254] For example, in a specific implementation, the image parsing unit first extracts the data values ​​of each pixel in the image to be processed, and calculates the grayscale changes of adjacent pixels along the row and column directions respectively to obtain the corresponding first rate of change and second rate of change, and then compares the two.

[0255] Specifically, when calculating the first rate of change, one can calculate the grayscale gradient value of one of the multiple pixel rows in the image to be processed and use it directly as the first rate of change, or one can calculate the grayscale gradient values ​​of each pixel row in the multiple pixel rows in the image to be processed and then calculate the average value to obtain the first rate of change.

[0256] When calculating the second rate of change, one can calculate the grayscale gradient value of one of the multiple pixel columns in the image to be processed and use it directly as the second rate of change, or one can calculate the grayscale gradient values ​​of each pixel column in the multiple pixel columns in the image to be processed and then calculate the average value to obtain the second rate of change.

[0257] The grayscale gradient value can be obtained by averaging the absolute values ​​of the differences between the grayscale values ​​of two adjacent pixels (horizontally or vertically adjacent).

[0258] In some cases, when the change in grayscale value in the row direction is no greater than the change in grayscale value in the column direction, it indicates that the image has a more stable distribution of adjacent data in the row direction. In this case, the data values ​​are arranged according to the first order. When the change in grayscale value in the row direction is greater than the change in grayscale value in the column direction, the data values ​​are arranged according to the second order. The first order corresponds to extracting pixel data values ​​row by row in the first direction and then expanding them in the second direction. The second order corresponds to extracting pixel data values ​​column by column in the second direction and then expanding them in the first direction.

[0259] In one possible embodiment, the above-mentioned calculation and comparison process of the rate of change can be completed by an image processor in conjunction with a parsing program in memory. The parsing program can be deployed in an embedded controller, a general-purpose processor, or a dedicated hardware acceleration unit. In practical applications, other models of processors can also be selected, and this application does not limit this.

[0260] In the example above, by aligning the arrangement of the data sequence to be processed with the direction of smaller grayscale changes in the image, the fluctuation range of adjacent data values ​​in the sequence can be further compressed to a smaller range. Subsequently, when determining the header data, the difference sequence, and the encoded values ​​corresponding to the differences, a more concentrated difference distribution can be formed, thereby reducing the storage redundancy of the target data sequence. Dynamically selecting the optimal data extraction method reduces the difference range, further improving the data compression rate and thus reducing the storage space occupied.

[0261] In some cases, image data processing can be a batch process. For example, a single processing batch may contain multiple image data, and the data processing method provided in the embodiments of this application is used to compress these multiple image data in the processing batch.

[0262] Therefore, in one example, there are multiple images to be processed.

[0263] For example, in practical application scenarios, when performing wafer defect detection, multiple photos may be taken simultaneously for the same wafer. These multiple photos of the same wafer can be used as multiple images to be processed in a single processing batch.

[0264] Based on this, Figure 8 Flowchart of the data processing method provided in this application Figure 6 ,like Figure 8 As shown, after arranging the data values ​​according to a first or second order to obtain the data sequence to be processed, the method further includes:

[0265] Step 801. Arrange the data values ​​of each pixel in one of the multiple images to be processed in a first order to obtain a first data sequence to be processed, and arrange them in a second order to obtain a second data sequence to be processed.

[0266] Step 802. For the first data sequence to be processed and the second data sequence to be processed, respectively execute the following steps: determine the header data and the difference sequence based on the data sequence to be processed; determine the encoding value corresponding to the difference based on the number of times the difference falls into the numerical range; and construct the target data sequence based on the header data, the difference, and the encoding value corresponding to the difference, to obtain the corresponding first target data sequence and the second target data sequence.

[0267] Step 803. Calculate the compression ratios of the first target data sequence and the second target data sequence respectively.

[0268] Compression ratio represents the rate of decrease in storage space occupied by the target data sequence relative to the data sequence to be processed.

[0269] Step 804. The splitting order corresponding to the higher compression ratio between the two is used as the splitting order for other images to be processed among the multiple images to be processed. The splitting order can be either the first order or the second order.

[0270] For example, in specific implementation, one of the multiple images to be processed is first unfolded in a first order and a second order to obtain two sets of data sequences to be processed. Then, the header data is extracted from the two sets of sequences and the adjacent differences are calculated. Subsequently, the corresponding encoding value is determined according to the number of times the difference falls into different numerical ranges. The header data, the difference and the encoding value are combined to form the target data sequence.

[0271] Subsequently, the storage space changes of the two target data sequences relative to the original data sequence to be processed are statistically analyzed to obtain their respective compression ratios. The arrangement corresponding to the higher compression ratio is then used as the unified splitting order for multiple subsequent images to be processed. This splitting order can be directly reused for other images, allowing subsequent images to use a more suitable arrangement under similar data distributions. Specifically, this splitting order is either the first order or the second order mentioned above.

[0272] The process of calculating the compression ratio of the two target data sequences can be referred to the method for calculating the compression ratio in the previous embodiment.

[0273] In the above example, by constructing target data sequences for a single image in two different orders and comparing their compression rates, and then extending the splitting order corresponding to the higher compression rate to other images to be processed, the arrangement method used in batch image processing can be matched with the data difference distribution. This reduces storage redundancy in subsequent images and ensures consistency and reusability of the compression results. When processing multiple images in a batch, the splitting direction with the higher compression rate can be determined in a single execution, ensuring a high compression rate when processing other images in the same batch.

[0274] The data processing method provided in this application can transform the original data to be processed into a difference sequence with smaller values, requiring fewer bits for storage and being easier to compress, by performing a difference operation. Furthermore, by preserving the header data to ensure the integrity of the difference sequence, a baseline value is provided for subsequent decompression. Finally, since the encoded value of the difference is negatively correlated with its frequency of occurrence, differences that occur frequently can be encoded with fewer bits, improving the compression ratio.

[0275] This application also provides a data storage method, including the following steps:

[0276] Perform the data processing method provided in any of the above embodiments or examples to construct a target data sequence and store the target data sequence.

[0277] This application also provides a data transmission method, including the following steps:

[0278] Perform the data processing method provided in any of the above embodiments or examples to construct a target data sequence and transmit the target data sequence.

[0279] The aforementioned data storage and data transmission methods, by constructing a target data sequence, can reduce the space occupied during storage, reduce the amount of data transmitted during transmission, and improve transmission efficiency.

[0280] This application also provides a data processing apparatus. This data processing apparatus is configured to perform the method provided in the above-described method embodiments.

[0281] In one possible implementation, Figure 9 A schematic diagram of the structure of the data processing apparatus provided in this application is shown below. Figure 9 As shown, the data processing apparatus 90 provided in this embodiment includes:

[0282] The acquisition module 901 is used to acquire a sequence of data to be processed; wherein, the sequence of data to be processed includes multiple data to be processed;

[0283] The processing module 902 is used to determine the header data and the difference sequence based on the data sequence to be processed; wherein, the header data is the first data to be processed in the data sequence to be processed; the difference sequence includes multiple differences, each of which is the difference between two adjacent data to be processed in the multiple data to be processed;

[0284] The processing module 902 is further configured to determine the encoding value corresponding to the difference based on the number of times the difference falls into the numerical range; wherein, the number of occurrences is the number of differences that fall into each numerical range among multiple differences; the encoding value is used to indicate the number of bits stored in the difference, and the number of bits in the encoding value is negatively correlated with the number of occurrences;

[0285] The processing module 902 is also used to construct the target data sequence based on the header data, the difference, and the encoded value corresponding to the difference.

[0286] In one possible implementation, the encoding value corresponding to the difference is determined based on the frequency of occurrences of the numerical range into which the difference falls. The processing module 902 is used to:

[0287] Based on multiple differences, determine the frequency of occurrence of different numerical ranges;

[0288] Based on the frequency of occurrence of each numerical range, different numerical ranges are grouped to obtain N numerical range groups; where each numerical range group includes at least one numerical range, and N is an integer greater than 1.

[0289] For differences that fall within the numerical range, set the encoding value corresponding to the numerical range group to which the numerical range belongs, so as to obtain the encoding value corresponding to the difference.

[0290] In one possible implementation, before determining the frequency of occurrence of different numerical ranges based on multiple differences, the processing module 902 is further configured to:

[0291] Determine the total number of occurrences for each value range group. The total number of occurrences is the sum of the number of occurrences of each value range included in the value range group.

[0292] The encoding value corresponding to the numerical range group is determined based on the total number of occurrences of the numerical range group; wherein, the encoding value corresponding to the numerical range group indicates the maximum number of bits required for storage in the numerical range group; the number of bits of the encoding value increases as the total number of occurrences of the numerical range group decreases.

[0293] In one possible implementation, the different numerical ranges are grouped according to the frequency of occurrence of each numerical range to obtain N numerical range groups. The processing module 902 is used for:

[0294] Determine the range of the first value of N-1 groups, where the range of the first value of N-1 groups is the first N-1 value range in the first sequence, and the first sequence is obtained by sorting the different value ranges in descending order of the frequency of occurrence.

[0295] To determine the first N-1 value range groups from N value range groups, the steps for determining each value range group in the first N-1 value range groups include: starting from the first value range of the i-th group in the second sequence, searching in reverse order of the second sequence; if other first value ranges of other groups are found, then the first value range of the i-th group and the value range between the first value range of the i-th group and the other first value ranges of other groups are taken as the i-th value range group; otherwise, the first value range of the i-th group, the first value range in the second sequence, and the value range between the first value range of the i-th group and the first value range in the second sequence are taken as the i-th value range group; wherein, the second sequence is obtained by sorting the different value ranges in ascending order according to the difference they represent; the initial value of i is 1, and i is an integer less than or equal to N-1;

[0296] Determine the Nth value range group among N value range groups, where the Nth value range group includes value ranges that do not belong to the first N-1 value range groups.

[0297] In one possible implementation, before determining the frequency of occurrence of different numerical ranges based on multiple differences, the processing module 902 is further configured to:

[0298] By selecting different values ​​of N, the steps of determining the encoding value corresponding to the difference based on the frequency of occurrence of the numerical range in which the difference falls are executed; and constructing the target data sequence based on the header data, the difference, and the encoding value corresponding to the difference are performed to obtain the target data sequence corresponding to different values ​​of N.

[0299] Based on the target data sequences corresponding to different values ​​of N, the compression ratio corresponding to each target data sequence is determined; whereby the compression ratio characterizes the rate of decrease in storage space occupied by the target data sequence relative to the data sequence to be processed;

[0300] The value of N corresponding to the highest compression ratio for each target data sequence is taken as the target value of N.

[0301] In one possible implementation, the target data sequence further includes: decoded information;

[0302] The decoding information includes the number of bits corresponding to each value range group in descending order of the total number of occurrences.

[0303] In one possible implementation, the acquisition module 901 is used to acquire the data sequence to be processed, and to:

[0304] The image to be processed is acquired and parsed to obtain the data values ​​of each pixel in the image.

[0305] Arrange the data values ​​in either the first or second order to obtain the data sequence to be processed;

[0306] The first sequence represents extracting multiple pixel rows from the image to be processed sequentially in a first direction, and extracting the data values ​​of multiple pixels in each pixel row sequentially in a second direction; the second sequence represents extracting multiple pixel columns from the image to be processed sequentially in a second direction, and extracting the data values ​​of multiple pixels in each pixel column sequentially in a first direction; the first direction is from top to bottom or from bottom to top; the second direction is from left to right or from right to left.

[0307] In one possible implementation, before arranging the data values ​​according to a first order or a second order to obtain the data sequence to be processed, the processing module 902 is further configured to:

[0308] For the image to be processed, a first rate of change and a second rate of change are determined; wherein, the first rate of change is the gray-level gradient value of the pixel row in the image to be processed; and the second rate of change is the gray-level gradient value of the pixel column in the image to be processed.

[0309] If the first rate of change is determined to be less than or equal to the second rate of change, the data values ​​are arranged in the first order to obtain the data sequence to be processed; otherwise, the data values ​​are arranged in the second order to obtain the data sequence to be processed.

[0310] In one possible implementation, there are multiple images to be processed. After arranging the data values ​​according to a first order or a second order to obtain a sequence of data to be processed, the processing module 902 is further configured to:

[0311] The data values ​​of each pixel in one of the multiple images to be processed are arranged in a first order to obtain a first data sequence to be processed, and arranged in a second order to obtain a second data sequence to be processed.

[0312] For the first and second data sequences to be processed, the following steps are performed respectively: determining the header data and difference sequence based on the data sequences to be processed; determining the encoding value corresponding to the difference based on the number of times the difference falls within the numerical range; and constructing the target data sequence based on the header data, the difference, and the encoding value corresponding to the difference, thereby obtaining the corresponding first and second target data sequences.

[0313] Calculate the compression ratios for the first target data sequence and the second target data sequence, respectively; where the compression ratio represents the rate of decrease in storage space occupied by the target data sequence relative to the data sequence to be processed;

[0314] The splitting order corresponding to the higher compression ratio between the two is used as the splitting order of other images to be processed among multiple images to be processed; wherein, the splitting order is either the first order or the second order.

[0315] Each of the above modules or units can be implemented in software. For example, the acquisition module 901 and the processing module 902 can both be implemented in software.

[0316] In this application, "implemented through software" means that the processor reads and executes program instructions stored in memory to implement the functions corresponding to the aforementioned modules or units. Here, the processor refers to a processing circuit capable of executing program instructions, including but not limited to at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and other processing circuits capable of running program instructions. In other embodiments, the processor may also include circuits with other processing functions (such as hardware circuits for hardware acceleration, bus and interface circuits, etc.). The processor can be presented as an integrated chip, for example, as an integrated chip whose processing function only includes executing software instructions, or it can also be presented as a SoC (system on a chip), that is, on a single chip, in addition to the processing circuit capable of running program instructions (referred to as the "core"), it also includes other hardware circuits for implementing specific functions (of course, these hardware circuits can also be implemented separately based on ASIC or FPGA). Correspondingly, the processing functions, in addition to executing software instructions, may also include various hardware acceleration functions (such as AI calculation, encoding / decoding, compression / decompression, etc.).

[0317] The division of modules is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0318] The data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0319] Figure 10 A schematic diagram of the structure of the electronic device provided in this application. Figure 10 As shown, the electronic device 100 provided in this embodiment includes at least one processor 1001 and a memory 1002. Optionally, the electronic device 100 further includes a communication component 1003. The processor 1001, memory 1002, and communication component 1003 are connected via a bus 1004.

[0320] Optionally, there may be one or more processors 1001.

[0321] Optionally, the memory 1002 may be one or more.

[0322] Alternatively, the memory 1002 can be integrated with the processor 1001, or it can be set separately.

[0323] In specific implementation, memory 1002 is used to store computer programs or instructions and / or data, and processor 1001 is used to execute the computer programs or instructions stored in memory 1002, or to read the data stored in memory 1002, so as to execute the methods in the above method embodiments.

[0324] The specific implementation process of processor 1001 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0325] It should be understood that the processor in the embodiments of this application can be an integrated circuit chip with the ability to process signals. In implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware encoding processor, or implemented by a combination of hardware and software modules in the encoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0326] The memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), and synchronous link dynamic memory (SLDRAM).

[0327] The bus in the embodiments of this application can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses in the accompanying drawings of this application are not limited to only one bus or one type of bus.

[0328] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory can be integrated into the processor.

[0329] It should also be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0330] This application also provides a data structure. This data structure includes header data, a difference, and the encoded value corresponding to the difference.

[0331] The header data is the first data to be processed in the data sequence; the difference is the difference between two adjacent data to be processed in the data sequence; the encoding value corresponding to the difference is used to indicate the number of bits stored in the difference, and the number of bits of the encoding value is negatively correlated with the number of occurrences of the numerical range to which the difference belongs.

[0332] In one possible implementation, the data structure further includes: decoded information;

[0333] The decoding information includes the number of bits corresponding to the total number of occurrences of the numerical range group in descending order; the numerical range group includes at least one numerical range, which is the numerical range into which the difference value falls.

[0334] For example, after obtaining the sequence of data to be processed, the first data to be processed in the sequence is used as the header data. Based on each data to be processed, the difference between two adjacent data is calculated to obtain multiple difference values, forming a difference sequence. The encoded value corresponding to each difference is determined based on the frequency of occurrences within the numerical range to which it falls. The number of bits in the encoded value of the difference decreases as the frequency of occurrences within its numerical range increases, and the encoded value indicates the actual number of bits used to store the difference.

[0335] Furthermore, based on the numerical range groups obtained by dividing the numerical range, the corresponding decoding bits of the encoded values ​​are set as decoding information, and set as a separate field between the data header and the encoded value of the difference in the data structure, according to the order of the total occurrence frequency of the numerical range groups from largest to smallest.

[0336] After obtaining the above header data, difference, difference encoded value and decoding information, the target data sequence is constructed in the order of data header, decoding information, the encoded value corresponding to the difference and the difference sequence (including multiple differences).

[0337] For example, the header data is 2 bytes long, the decoded information is 4 bits × 4, the length of the encoded value corresponding to the difference is 512-512 × 3 bits, and the length of the difference sequence is 512-512 × 16 bits.

[0338] By performing subtraction, the original data to be processed can be transformed into a difference sequence with smaller values, requiring fewer bits for storage and thus being easier to compress. Furthermore, by preserving the header data to ensure the integrity of the difference sequence, a baseline value is provided for subsequent decompression. Finally, since the encoded value of the difference is negatively correlated with its frequency of occurrence, differences that occur more frequently can be encoded with fewer bits, improving the compression ratio. It should be noted that the specific implementation methods and technical effects can be referred to the foregoing method embodiments.

[0339] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0340] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0341] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0342] This application also provides a computer program product, which includes computer program code or instructions, such that when the computer program code or instructions are run on a computer, the methods in the various method embodiments of this application are executed.

[0343] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0344] Furthermore, this application also provides a chip including a processor. A memory for storing a computer program is provided independently of the chip, and the processor is used to execute the computer program stored in the memory, so that the methods in the various method embodiments of this application are performed.

[0345] Furthermore, the chip may also include a communication interface. The communication interface can be an input / output interface or an interface circuit, etc. Furthermore, the chip may also include memory.

[0346] The foregoing preferred embodiments have further detailed the purpose, technical solutions, and advantages of this application. It should be understood that the above descriptions are merely preferred embodiments and are not intended to limit the application. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the scope of protection of this application. Finally, it should be noted that those skilled in the art, upon considering the specification and practicing the invention disclosed herein, will readily conceive of other embodiments of the invention. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. It is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A data processing method, characterized in that, include: Obtain a sequence of data to be processed; wherein the sequence of data to be processed includes multiple data items to be processed; Based on the data sequence to be processed, a header data and a difference sequence are determined; wherein, the header data is the first data to be processed in the data sequence to be processed; the difference sequence includes multiple differences, each of which is the difference between two adjacent data to be processed in the multiple data to be processed. The encoding value corresponding to the difference is determined based on the number of times the difference falls within the numerical range; wherein, the number of occurrences is the number of differences among the plurality of differences that fall within each of the numerical ranges; the encoding value is used to indicate the number of bits stored for the difference, and the number of bits of the encoding value is negatively correlated with the number of occurrences; The target data sequence is constructed based on the header data, the difference, and the encoded value corresponding to the difference.

2. The method according to claim 1, characterized in that, The encoded value corresponding to the difference is determined based on the frequency of occurrence of the numerical range within which the difference falls, including: Based on the multiple differences, determine the frequency of occurrence of different numerical ranges; Based on the frequency of occurrence of each numerical range, the different numerical ranges are grouped to obtain N numerical range groups; wherein each numerical range group includes at least one numerical range, and N is an integer greater than 1. For the difference that falls within the range of values, set the encoding value corresponding to the value range group to which the value range belongs, so as to obtain the encoding value corresponding to the difference.

3. The method according to claim 2, characterized in that, Before determining the frequency of occurrence of different numerical ranges based on the multiple differences, the method further includes: Determine the total number of occurrences for each value range group, where the total number of occurrences is the sum of the number of occurrences of each value range included in the value range group; The encoding value corresponding to the numerical range group is determined based on the total number of occurrences of the numerical range group; wherein the encoding value corresponding to the numerical range group indicates the maximum number of bits required for storage in the numerical range group; the number of bits of the encoding value increases as the total number of occurrences of the numerical range group decreases.

4. The method according to claim 2 or 3, characterized in that, Based on the frequency of occurrence of each numerical range, the different numerical ranges are grouped to obtain N numerical range groups, including: Determine the first value range of N-1 groups, wherein the first value range of N-1 groups is the first N-1 value range in the first sequence, and the first sequence is obtained by sorting the different value ranges in descending order of the frequency of occurrence; The method for determining the first N-1 value range groups from the N value range groups includes: starting from the first value range of the i-th group in the second sequence, searching in reverse order of the second sequence; if other first value ranges of groups are found, then the first value range of the i-th group and the value range between the first value range of the i-th group and the other first value ranges of groups are taken as the i-th value range group; otherwise, the first value range of the i-th group, the first value range in the second sequence, and the value range between the first value range of the i-th group and the first value range in the second sequence are taken as the i-th value range group; wherein, the second sequence is obtained by sorting the different value ranges in ascending order according to the difference represented by them; the initial value of i is 1, and i is an integer less than or equal to N-1; Determine the Nth numerical range group among the N numerical range groups, wherein the Nth numerical range group includes numerical ranges that do not belong to the first N-1 numerical range groups.

5. The method according to any one of claims 2-4, characterized in that, Before determining the frequency of occurrence of different numerical ranges based on the multiple differences, the method further includes: By selecting different values ​​of N, the steps of determining the encoding value corresponding to the difference based on the frequency of occurrence of the numerical range in which the difference falls are executed; and the steps of constructing a target data sequence based on the header data, the difference, and the encoding value corresponding to the difference are executed respectively, to obtain the target data sequence corresponding to different values ​​of N; Based on the target data sequences corresponding to different values ​​of N, the compression ratio corresponding to each target data sequence is determined; wherein, the compression ratio characterizes the rate of decrease in storage space occupied by the target data sequence relative to the data sequence to be processed; The value of N corresponding to the highest compression ratio for each of the target data sequences is taken as the target value of N.

6. The method according to any one of claims 2-5, characterized in that, The target data sequence also includes: decoded information; The decoding information includes the number of decoding bits corresponding to the total number of occurrences of the numerical range group in descending order.

7. The method according to any one of claims 1-6, characterized in that, The acquisition of the data sequence to be processed includes: The image to be processed is acquired and parsed to obtain the data values ​​of each pixel in the image to be processed. The data values ​​are arranged in a first order or a second order to obtain the data sequence to be processed; Wherein, the first order represents extracting multiple pixel rows from the image to be processed sequentially in a first direction, and extracting data values ​​of multiple pixels from each pixel row sequentially in a second direction; the second order represents extracting multiple pixel columns from the image to be processed sequentially in a second direction, and extracting data values ​​of multiple pixels from each pixel column sequentially in a first direction; the first direction is from top to bottom or from bottom to top; the second direction is from left to right or from right to left.

8. The method according to claim 7, characterized in that, Before arranging the data values ​​according to a first order or a second order to obtain the data sequence to be processed, the method further includes: For the image to be processed, a first rate of change and a second rate of change are determined; wherein, the first rate of change is the gray-level gradient value of the pixel row in the image to be processed; and the second rate of change is the gray-level gradient value of the pixel column in the image to be processed. If the first rate of change is determined to be less than or equal to the second rate of change, the data values ​​are arranged in a first order to obtain the data sequence to be processed; otherwise, the data values ​​are arranged in a second order to obtain the data sequence to be processed.

9. The method according to claim 7, characterized in that, The number of images to be processed is multiple. After arranging the data values ​​according to a first order or a second order to obtain the data sequence to be processed, the method further includes: The data values ​​of each pixel in one of the multiple images to be processed are arranged in a first order to obtain a first data sequence to be processed, and arranged in a second order to obtain a second data sequence to be processed. For the first data sequence to be processed and the second data sequence to be processed, respectively, the following steps are performed: determining the header data and the difference sequence based on the data sequence to be processed; determining the encoding value corresponding to the difference based on the number of times the difference falls into the numerical range; and constructing the target data sequence based on the header data, the difference, and the encoding value corresponding to the difference, to obtain the corresponding first target data sequence and the second target data sequence. Calculate the compression ratios for the first target data sequence and the second target data sequence, respectively; wherein the compression ratio represents the rate of decrease in storage space occupied by the target data sequence relative to the data sequence to be processed; The splitting order corresponding to the higher compression ratio between the two is used as the splitting order of other images to be processed among the plurality of images to be processed; wherein, the splitting order is either the first order or the second order.

10. A data processing apparatus, characterized in that, It is configured to perform the method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-9.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

Images