Data processing method and device, storage system and computer readable medium

Abstract

The invention discloses a data processing method and device, a storage system and a computer readable medium, relates to the technical field of data processing, and solves the problems that effective consideration on different types of data conversion time synchronism is lacked, and conversion computing power cannot be accurately distributed according to data characteristics. According to the method, the corresponding number of conversion threads are executed based on the characteristics of different types of data in the total data packet, and proper computing power is distributed to each thread. Meanwhile, the conversion amount and the conversion rate of the thread are monitored in real time, and the computing power is dynamically readjusted according to the actual situation. According to the mechanism, the problem of unbalanced conversion processes caused by factors such as data retrieval speed difference can be effectively solved, it is ensured that each conversion thread can run in a relatively balanced state, the synchronism of data conversion is further improved, the whole data conversion process is more efficient and reliable, and the user experience is improved. And the condition that the overall processing time is prolonged due to individual thread delay is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method, device, storage system and computer-readable medium. Background Art

[0002] In today's digital age, data is growing explosively, and various data processing tasks are becoming increasingly arduous and complex. Data sources are extensive, covering multiple channels such as business systems, sensor networks, and the Internet, and their types are also rich and varied, including text, images, audio, etc. In the data processing process, data conversion is one of the key links.

[0003] As the demand for data processing continues to increase, traditional data processing methods face many challenges. During the data conversion stage, different types of data have their own unique structures and encoding characteristics. For example, text data is based on character encoding, image data is closely related to the binary representation of pixels, and audio data has its own specific audio feature encoding; these differences result in very different processing time and computing resources required when converting to binary data that can be processed by computers.

[0004] Previous data processing technologies often lacked effective consideration of the time synchronization of conversion of different types of data, and failed to accurately allocate conversion computing power based on data characteristics. This made it very easy for the data conversion of the current cycle to be completed before the next cycle data had already begun to be converted in actual operations. This data chaos will not only lead to problems such as data errors and loss, seriously affecting the accuracy of data processing, but will also cause the entire data processing process to fall into disorder, reduce processing efficiency, increase the time cost and resource consumption of data processing, and cannot meet the urgent needs of modern society for efficient and accurate data processing. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a data processing method, device, storage system and computer-readable medium, which solve the problems of lack of effective consideration of the time synchronization of conversion of different types of data and failure to accurately allocate conversion computing power according to data characteristics.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a data processing method, comprising the following steps:

[0007] Step 1: Confirm the total data package that needs to be processed at the current stage. The total data package is input by relevant personnel or data is collected from different ports that are set;

[0008] Step 2: Differentiate different types of data in the data package, and from the differentiated different types of data, confirm the total amount of features associated with the different types of data, confirm the total amount of binary features associated with the corresponding type of data based on the confirmed total amount of features and the associated binary features, and divide the conversion computing power based on the confirmed total amount of binary features of the different types of data to determine the average computing power associated with the corresponding type of data;

[0009] Step 3: Based on the total number of different types of data in the total data package, the same number of conversion threads are executed, each conversion thread performs binary conversion on a group of type data to obtain binary conversion data of the corresponding type data, and the execution computing power of the conversion thread associated with the corresponding type data is the confirmed average computing power, the actual conversion amount and conversion rate of each thread are monitored in real time, and the real-time monitored data are analyzed in real time, and the average computing power associated with each thread is readjusted. The specific sub-steps are:

[0010] S31, enabling different conversion threads to perform binary conversion at the same time, and monitoring the remaining capacity of data to be converted in different conversion threads in real time, and marking the monitored capacity as R k , where k represents different conversion threads, and then based on the conversion rate J associated with the current moment in the corresponding conversion thread k ;

[0011] S32, based on the confirmed R k and J k , using: T k =R k ÷J k Confirm the conversion time T associated with the corresponding conversion thread k , the different conversion times T associated with different conversion threads k Perform standard deviation processing to confirm the total processing characteristics CT;

[0012] Identify whether the confirmed total processing feature CT satisfies: CT≤Y1, where Y1 is a preset value. If it does, continue monitoring. If it does not, perform computing power scheduling from the conversion thread associated with the shortest conversion time to the conversion thread associated with the longest conversion time, and confirm the total processing features associated with subsequent moments until the confirmed total processing feature CT≤Y1 is reached, and computing power scheduling is stopped. A group of unit computing power is scheduled at each different moment, and the unit computing power is the preset computing power. During the computing power scheduling process, the conversion thread associated with the shortest conversion time or the conversion thread associated with the longest conversion time will change during the scheduling process. Continue monitoring after stopping computing power scheduling.

[0013] Step 4: Read and write data for the total data package after data conversion is completed. From the historical data of different binary conversion data, confirm the different read and write rates associated with different CPU usage values, and then based on the specific usage that the current CPU can use, evenly distribute this specific usage to the read and write processes of different binary conversion data, so that multiple different binary conversion data can synchronously complete the read, write and storage process within the standard time difference. The specific sub-steps are:

[0014] S41, mark the specific proportion that the current CPU can use as ZB, and then prioritize several groups of read and write rates associated with the CPU proportion from the historical data of different binary conversion data, average the several groups of read and write rates, determine the average rate and mark it as V q , where q represents different CPU usage rates, and then the different mean rates V associated with different CPU usage rates for this binary conversion data q Confirm in sequence and generate a mean rate change curve belonging to the binary conversion data. The horizontal axis of the mean rate change curve is the CPU percentage, and the vertical axis is the mean rate.

[0015] S42, confirm the average rate change curves of several binary conversion data that currently need to be read and written in sequence, and based on the determined specific proportion ZB, randomly allocate this ZB, execute several allocation processes, and confirm the time characteristics associated with each allocation process:

[0016] The CPU ratio allocated to the corresponding binary conversion data in the corresponding allocation process is marked as C g , where g represents different binary conversion data, and then from the mean rate change curve associated with this binary conversion data, confirm C g The associated mean rate V g , then based on the total amount of data R corresponding to the binary conversion data g , confirm the read and write time associated with its binary conversion data: its read and write time = R g ÷V g ;

[0017] Confirm the read and write times associated with different binary conversion data in the allocation process in turn, and use the several read and write times associated with the allocation process as the time characteristics of the allocation process;

[0018] S43, based on the different time features associated with different allocation processes, a number of read and write times corresponding to the time features are processed for variance, a calibration variance is confirmed, and it is identified whether the calibration variance associated with the corresponding allocation process satisfies: calibration variance ≤ Y2, where Y2 is a preset value, which is a preset standard time difference. If so, the corresponding allocation process is calibrated as a qualified process;

[0019] S44. From the confirmed several qualified processes, a group of qualified processes with the smallest calibration variance is selected as the selected processes, and the CPU share allocated to different binary conversion data in the selected process is used as the execution share, and the specific share ZB is directly allocated, and the data reading and writing processes corresponding to the several binary conversion data are directly executed, and the several binary conversion data are synchronously read and written at the same time.

[0020] Preferably, a group of data packets is confirmed every time a group of collection periods is set, and the collection period is a preset period.

[0021] Preferably, if the calibration variance>Y2, no calibration is performed on the allocation process.

[0022] Preferably, a data processing device comprises:

[0023] The data package confirmation end confirms the data package that needs to be processed at the current stage. The data package is input by relevant personnel or data is collected from different ports that are set. A group of data packages is confirmed every group of collection cycles.

[0024] The computing power equalization processing end distinguishes different types of data in the data package, confirms the total amount of features associated with the different types of data from the distinguished different types of data, confirms the total amount of binary associated with the corresponding type of data based on the confirmed total amount of features and the associated binary features, and performs equalization of conversion computing power based on the confirmed total amount of binary of the different types of data to determine the equalized computing power associated with the corresponding type of data;

[0025] The real-time control end of the conversion process executes the same number of conversion threads based on the total number of different types of data in the total data package. Each conversion thread performs binary conversion on a group of type data to obtain binary conversion data of the corresponding type data, and the execution computing power of the conversion thread associated with the corresponding type data is the confirmed average computing power. The actual conversion amount and conversion rate of each thread are monitored in real time, and the real-time monitored data are analyzed in real time to readjust the average computing power associated with each thread.

[0026] The data reading and writing synchronization end reads and writes the total data package after the data conversion is completed. From the historical data of different binary conversion data, the different reading and writing rates associated with different CPU usage ratio values ​​are confirmed, and then based on the specific usage ratio that the current CPU can use, this specific usage ratio is evenly distributed to the reading and writing processes of different binary conversion data, so that multiple different binary conversion data can synchronously complete the reading, writing and storage processes within the standard time difference.

[0027] Preferably, a storage system stores data packets that complete data reading, writing and storage processes at different stages, and stores different binary conversion data in the data packets in different storage partitions.

[0028] Preferably, a computer readable medium internally stores a plurality of sets of preset values, including a preset value Y1, for comparing the confirmed total processing feature CT;

[0029] A preset value Y2 is also included for comparing the confirmed calibration variance.

[0030] The present invention provides a data processing method, device, storage system and computer-readable medium. Compared with the prior art, the present invention has the following beneficial effects:

[0031] Execute a corresponding number of conversion threads based on the characteristics of different types of data in the data package, and allocate appropriate computing power to each thread. At the same time, monitor the conversion volume and conversion rate of the thread in real time, and dynamically readjust the computing power according to the actual situation; this mechanism can effectively deal with the imbalance of the conversion process caused by factors such as differences in data retrieval speed, ensure that each conversion thread can run in a relatively balanced state, further improve the synchronization of data conversion, make the entire data conversion process more efficient and reliable, and reduce the overall processing time extension caused by individual thread delays;

[0032] In the data reading and writing stage, the impact of CPU usage on the reading and writing rate is fully considered. The average reading and writing rate under different CPU usage rates is determined by analyzing historical data, and the average rate change curve is generated. Based on this, the current CPU available usage rate is reasonably allocated. After multiple rounds of testing and screening of the allocation process, an allocation scheme is finally selected that enables multiple binary conversion data to complete the reading and writing storage process synchronously within the standard time difference; this not only improves the speed of data reading and writing, but also ensures the synchronization of reading and writing of different types of binary data, avoids the occurrence of data reading and writing retention, effectively improves the efficiency and reliability of data storage, provides strong support for subsequent data applications, and ensures the smooth operation of the entire data processing system and the integrity of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the process of the present invention;

[0034] Figure 2 It is a schematic diagram of the framework principle of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] First embodiment

[0037] See also Figure 1 , the present application provides a data processing method, comprising the following steps:

[0038] Step 1: Confirm the total data package that needs to be processed at the current stage. The total data package is input by relevant personnel or data is collected from different ports that are set. A group of total data packages is confirmed every time a group of collection cycles is set. The collection cycle is a preset cycle, which is prepared by relevant operators based on experience. The total data package is the data packet that needs to be processed, and the data packet includes different data from different source ports, that is, the data packet includes different data of different types;

[0039] Step 2: Distinguish different types of data in the data package, confirm the total amount of features associated with different types of data from the distinguished different types of data, confirm the total amount of binary associated with the corresponding type of data based on the confirmed total amount of features and the associated binary features, and divide the conversion computing power equally based on the confirmed total amount of binary for different types of data to determine the equalized computing power associated with the corresponding type of data. Specifically, after a data package is generated, its data package may include different types of data, such as text data, image data, and audio data, wherein a characteristic text character in the text data corresponds to a set of 8-bit binary codes, and each pixel in the image data is associated with a set of 24-bit binary codes, and each pixel has In a set of RGB values, its R value, G value and B value all correspond to a set of 8-bit binary codes, so each pixel feature is associated with a set of 24-bit binary parameters, and each audio feature in the corresponding audio data is associated with a set of 16-bit binary parameters. When the computer processes the data, it needs to convert different data into recognizable binary parameters, and then perform read-write storage management. In order to make the conversion time more synchronized, it is necessary to identify the total amount of different binary data associated with different types of data, so as to perform better synchronization confirmation of the conversion time. The reason for synchronization is to avoid data confusion. If the time synchronization confirmation is not performed, the data of this cycle has not been converted yet, and the data of the next cycle will intervene in the conversion process, which can easily cause data confusion;

[0040] The specific sub-steps for converting different types of data and equally distributing the computing power are as follows:

[0041] S21. Confirm different types of data in different data formats in the data package (the data formats can be directly identified, such as text format, image format, etc.), and then confirm the total number of data features existing in the different types of data (text character features, image pixel features, etc.), and mark the total number of data features of the confirmed different types of data as Z. i , where i represents different types of data, and then confirm the binary features T associated with different types of data i , where binary features are preset features, which are formulated by relevant operators based on experience. Different data have different total number of binary features. The following is used: ZL i =Z i ×T i Confirm the binary total ZL associated with the corresponding type of data i ;

[0042] S22, based on different binary totals ZL associated with different types of data i , the confirmed binary total ZL iPerform ratio processing, confirm the binary total ratio sequence, sum the ratios at different positions within the binary total ratio sequence, confirm the total score ZF, and calibrate the ratio at the unit position as a single score D i (Ratios at different positions correspond to different types of data), using: (ZS÷ZF)×D i =SL i Confirm the average computing power SL obtained by evenly dividing the corresponding type of data i , where ZS is the conversion computing power contained in this processor (the corresponding computing power held in the original internal). For example: the original computing power is 100, and there are three different types of data in its total data package. The total binary volume associated with each different type of data is 200, 200, and 300 respectively. Then the determined binary volume ratio sequence is 2:2:3. Based on this ratio sequence, the total score is confirmed to be 7, so the computing power allocated to the first group of data is 28.5. Similarly, the average computing power associated with each different type of data is confirmed in turn;

[0043] Step 3: Based on the total number of different types of data in the data package, the same number of conversion threads are executed. Each conversion thread performs binary conversion on a group of data types to obtain binary conversion data of the corresponding type of data, and the execution computing power of the conversion thread associated with the corresponding type of data is the confirmed average computing power. The actual conversion amount and conversion rate of each thread are monitored in real time, and the real-time monitored data is analyzed in real time, and the average computing power associated with each thread is readjusted. Specifically, in the actual conversion process, each different process performs binary conversion on different types of data. During the conversion, due to the data retrieval speed of some data during the conversion, there will be a speed difference between the conversions, but this difference is not large. In order to effectively avoid this situation, the computing power associated with each process is fine-tuned to achieve a relatively balanced effect to ensure that the different types of data included in this data package can complete the data conversion at the same time. Among them, the specific sub-steps of readjusting the average computing power associated with each thread are:

[0044] S31, making different conversion threads perform binary conversion at the same time, and monitoring the capacity of the remaining data to be converted (that is, the corresponding type of data) in different conversion threads in real time, and marking the monitored capacity as R k , where k represents different conversion threads, and then based on the conversion rate J associated with the current moment in the corresponding conversion thread k ;

[0045] S32, based on the confirmed R k and J k , using: T k =R k÷J k Confirm the conversion time T associated with the corresponding conversion thread k (that is, the conversion time required for the conversion state according to this conversion rate), the different conversion times T associated with different conversion threads k Perform standard deviation processing and confirm the total processing characteristics CT. The standard deviation processing method is: set k = 1, 2, ..., n, and use To determine the corresponding total processing characteristics CT, where is the conversion time T of n groups k The mean of , where the standard deviation represents the dispersion between the corresponding times;

[0046] Identify whether the confirmed total processing feature CT satisfies: CT≤Y1, where Y1 is a preset value, and its specific value is determined by the operator in advance based on experience. If it satisfies, it can be continuously monitored (the corresponding conversion time is still specifically confirmed at different moments in the future). If it does not satisfy, the computing power is scheduled from the conversion thread associated with the shortest conversion time to the conversion thread associated with the longest conversion time, and the total processing features associated with the subsequent moments are confirmed until the confirmed total processing feature CT≤Y1 is stopped. A group of unit computing power is scheduled at each different moment, and its unit computing power is the preset computing power, which is a preset unit value. During the computing power scheduling process, the conversion thread associated with the shortest conversion time or the conversion thread associated with the longest conversion time will change in the scheduling process (that is, after the computing power scheduling, the conversion thread associated with the longest conversion time will change, and similarly, the conversion thread associated with the shortest conversion time will also change synchronously). Continue to monitor after stopping the computing power scheduling;

[0047] Specifically, in order to ensure the synchronization of several different conversion processes, it is necessary to confirm the total required time associated with different conversion processes, and make specific confirmation of the data duration based on the real-time conversion rate and the data capacity to be converted. Based on the specific process confirmed in real time, real-time adjustment is made, and computing power is scheduled from the fast processing side to the slow processing side to complete the specific scheduling process. In this way, the different conversion processes associated with each corresponding stage can complete the data conversion process in a relatively similar time, thus achieving relative synchronization.

[0048] Step 4: Read and write data (that is, data storage) for the total data package after the data conversion is completed. From the historical data of different binary conversion data, confirm the different read and write rates associated with different CPU share values, and then based on the specific share that the current CPU can use, evenly distribute this specific share to the read and write processes of different binary conversion data, so that multiple different binary conversion data can synchronously complete the read and write storage process within the standard time difference. Specifically, step three is mainly aimed at the synchronous conversion between different types of data, and step four is mainly aimed at the synchronous storage and reading and writing of different types of binary data, to ensure that multiple different binary data can complete the read and write process at the same time, to avoid the corresponding read and write data being retained in the next cycle, resulting in data confusion;

[0049] The specific sub-steps for evenly distributing this specific ratio to the reading and writing process of different binary conversion data are as follows:

[0050] S41. The specific utilization ratio of the current CPU is calibrated as ZB (because computers have different utilization conditions, the data reading and writing at each different time will be affected by the specific CPU utilization ratio. For example, when a certain software is updated, it will be stuck when executing another software, which affects the data reading and writing rate). Then, from the historical data of different binary conversion data, several groups of reading and writing rates associated with the CPU utilization ratio are preferentially confirmed, and the several groups of reading and writing rates are averaged, and the average rate is confirmed and calibrated as V. q , where q represents different CPU usage rates, and then the different mean rates V associated with different CPU usage rates for this binary conversion data q Confirm in sequence and generate a mean rate change curve belonging to the binary conversion data. The horizontal axis of the mean rate change curve is the CPU percentage, and the vertical axis is the mean rate.

[0051] S42, confirm the average rate change curves of several binary conversion data that currently need to be read and written in sequence, and based on the determined specific proportion ZB, randomly allocate this ZB, execute several allocation processes, and confirm the time characteristics associated with each allocation process:

[0052] The CPU ratio allocated to the corresponding binary conversion data in the corresponding allocation process is marked as C g , where g represents different binary conversion data, and then from the mean rate change curve associated with this binary conversion data, confirm C g The associated mean rate V g , then based on the total amount of data R corresponding to the binary conversion data g , confirm the read and write time associated with its binary conversion data: its read and write time = Rg ÷V g ;

[0053] Confirm the read and write times associated with different binary conversion data in the allocation process in turn, and use the several read and write times associated with the allocation process as the time characteristics of the allocation process;

[0054] S43, based on the different time features associated with different allocation processes, the corresponding read and write times in the time features are processed for variance, the calibration variance is confirmed, and it is identified whether the calibration variance associated with the corresponding allocation process satisfies: calibration variance ≤ Y2, where Y2 is a preset value, and its specific value is determined by the operator based on experience, which is a preset standard time difference. If it satisfies, the corresponding allocation process is calibrated as a qualified process, otherwise, no calibration is performed;

[0055] S44, from the confirmed several qualified processes, a group of qualified processes with the smallest calibration variance is selected as the selected processes, and the CPU proportions allocated to different binary conversion data in the selected processes are used as the execution proportions, and the specific proportions ZB are directly allocated, and the data reading and writing processes corresponding to the several binary conversion data are directly executed, and the several binary conversion data are synchronously read and written at the same time;

[0056] Specifically, when reading and writing data, each different CPU share has a different read and write rate, and thus a different read and write progress. In order to effectively and reasonably allocate the existing share, the different read and write rates associated with different share are confirmed from the historical completion data, and then the associated read and write duration is confirmed based on the total amount of data of the corresponding data. When multiple different binary data that need to be read and written synchronously can meet the standards in terms of read and write duration, it means that the corresponding allocated share is in a better situation, and a better processing effect can be achieved, so that multiple different data can complete the synchronous reading and writing process at the same time.

[0057] Second embodiment

[0058] Combination Figure 2 , a data processing device, comprising:

[0059] The data package confirmation end confirms the data package that needs to be processed at the current stage. The data package is input by relevant personnel or data is collected from different ports that are set. A group of data packages is confirmed every group of collection cycles.

[0060] The computing power equalization processing end distinguishes different types of data in the data package, confirms the total amount of features associated with the different types of data from the distinguished different types of data, confirms the total amount of binary associated with the corresponding type of data based on the confirmed total amount of features and the associated binary features, and performs equalization of conversion computing power based on the confirmed total amount of binary of the different types of data to determine the equalized computing power associated with the corresponding type of data;

[0061] The real-time control end of the conversion process executes the same number of conversion threads based on the total number of different types of data in the total data package. Each conversion thread performs binary conversion on a group of type data to obtain binary conversion data of the corresponding type data, and the execution computing power of the conversion thread associated with the corresponding type data is the confirmed average computing power. The actual conversion amount and conversion rate of each thread are monitored in real time, and the real-time monitored data are analyzed in real time to readjust the average computing power associated with each thread.

[0062] The data reading and writing synchronization end reads and writes the total data package after the data conversion is completed. From the historical data of different binary conversion data, the different reading and writing rates associated with different CPU usage ratio values ​​are confirmed, and then based on the specific usage ratio that the current CPU can use, this specific usage ratio is evenly distributed to the reading and writing processes of different binary conversion data, so that multiple different binary conversion data can synchronously complete the reading, writing and storage processes within the standard time difference.

[0063] A storage system stores data packets that complete data reading, writing and storage processes at different stages, and stores different binary conversion data in the data packets in different storage partitions;

[0064] A computer readable medium having a plurality of sets of preset values ​​stored therein, including a preset value Y1, for comparing the confirmed total processing feature CT;

[0065] A preset value Y2 is also included for comparing the confirmed calibration variance.

[0066] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0067] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A data processing method, characterized in that: The following steps are involved: Step 1: Confirm the total data package that needs to be processed at the current stage. The total data package is input by relevant personnel or data is collected from different ports that are set; Step 2: Differentiate different types of data in the data package, and from the differentiated different types of data, confirm the total amount of features associated with the different types of data, confirm the total amount of binary features associated with the corresponding type of data based on the confirmed total amount of features and the associated binary features, and divide the conversion computing power based on the confirmed total amount of binary features of the different types of data to determine the average computing power associated with the corresponding type of data; Step 3: Based on the total number of different types of data in the total data package, the same number of conversion threads are executed, each conversion thread performs binary conversion on a group of type data to obtain binary conversion data of the corresponding type data, and the execution computing power of the conversion thread associated with the corresponding type data is the confirmed average computing power, the actual conversion amount and conversion rate of each thread are monitored in real time, and the real-time monitored data are analyzed in real time, and the average computing power associated with each thread is readjusted; Step 4: Read and write data for the total data package after the data conversion is completed. From the historical data of different binary conversion data, confirm the different read and write rates associated with different CPU usage ratio values, and then based on the specific usage ratio that the current CPU can use, evenly distribute this specific usage ratio to the read and write processes of different binary conversion data, so that multiple different binary conversion data can synchronously complete the read, write and storage processes within the standard time difference.

2. A data processing method according to claim 1, characterized in that: In the step 1, a group of data packets is confirmed every group of collection periods, and the collection period is a preset period.

3. A data processing method according to claim 1, characterized in that: In step 2, the specific sub-steps of equally dividing the conversion computing power of different types of data are: S21, confirming different types of data belonging to different data formats in the data package, and then confirming the total number of data features existing in the different types of data, and marking the confirmed total number of data features of the different types of data as Z. i , where i represents different types of data, and then confirm the binary features T associated with different types of data i , where binary features are preset features, using: ZL i =Z i ×T i Confirm the binary total ZL associated with the corresponding type of data i ; S22, based on different binary totals ZL associated with different types of data i , the confirmed binary total ZL i Perform ratio processing, confirm the binary total ratio sequence, sum the ratios at different positions within the binary total ratio sequence, confirm the total score ZF, and calibrate the ratio at the unit position as a single score D i , using: (ZS÷ZF)×D i =SL i Confirm the average computing power SL obtained by evenly dividing the corresponding type of data i , where ZS is the conversion computing power included in this processor.

4. A data processing method according to claim 3, characterized in that: In step 3, the specific sub-steps for re-adjusting the average computing power associated with each thread are: S31, enabling different conversion threads to perform binary conversion at the same time, and monitoring the remaining capacity of data to be converted in different conversion threads in real time, and marking the monitored capacity as R k , where k represents different conversion threads, and then based on the conversion rate J associated with the current moment in the corresponding conversion thread k ; S32, based on the confirmed R k and J k , using: T k =R k ÷J k Confirm the conversion time T associated with the corresponding conversion thread k , the different conversion times T associated with different conversion threads k Perform standard deviation processing to confirm the total processing characteristics CT; Identify whether the confirmed total processing feature CT satisfies: CT≤Y1, where Y1 is a preset value. If it does, continue monitoring. If not, perform computing power scheduling from the conversion thread associated with the shortest conversion time to the conversion thread associated with the longest conversion time, and confirm the total processing features associated with subsequent moments until the confirmed total processing feature CT≤Y1, then stop computing power scheduling. Schedule a group of unit computing power at each different moment, and the unit computing power is the preset computing power. During the computing power scheduling process, the conversion thread associated with the shortest conversion time or the conversion thread associated with the longest conversion time will change during the scheduling process. Continue monitoring after stopping computing power scheduling.

5. A data processing method according to claim 1, characterized in that: In step 4, the specific sub-steps of evenly distributing the specific proportion to the reading and writing processes of different binary conversion data are as follows: S41, mark the specific proportion that the current CPU can use as ZB, and then prioritize several groups of read and write rates associated with the CPU proportion from the historical data of different binary conversion data, average the several groups of read and write rates, determine the average rate and mark it as V q , where q represents different CPU usage rates, and then the different mean rates V associated with different CPU usage rates for this binary conversion data q Confirm in sequence and generate a mean rate change curve belonging to the binary conversion data. The horizontal axis of the mean rate change curve is the CPU percentage, and the vertical axis is the mean rate. S42, confirm the average rate change curves of several binary conversion data that currently need to be read and written in sequence, and based on the determined specific proportion ZB, randomly allocate this ZB, execute several allocation processes, and confirm the time characteristics associated with each allocation process: The CPU ratio allocated to the corresponding binary conversion data in the corresponding allocation process is marked as C g , where g represents different binary conversion data, and then from the mean rate change curve associated with this binary conversion data, confirm C g The associated mean rate V g , then based on the total amount of data R corresponding to the binary conversion data g , confirm the read and write time associated with its binary conversion data: its read and write time = R g ÷V g ; Confirm the read and write times associated with different binary conversion data in the allocation process in turn, and use the several read and write times associated with the allocation process as the time characteristics of the allocation process; S43, based on the different time features associated with different allocation processes, a number of read and write times corresponding to the time features are processed for variance, a calibration variance is confirmed, and it is identified whether the calibration variance associated with the corresponding allocation process satisfies: calibration variance ≤ Y2, where Y2 is a preset value, which is a preset standard time difference. If so, the corresponding allocation process is calibrated as a qualified process; S44. From the confirmed several qualified processes, a group of qualified processes with the smallest calibration variance is selected as the selected processes, and the CPU share allocated to different binary conversion data in the selected process is used as the execution share, and the specific share ZB is directly allocated, and the data reading and writing processes corresponding to the several binary conversion data are directly executed, and the several binary conversion data are synchronously read and written at the same time.

6. A data processing method according to claim 5, characterized in that: In step S43, if the calibration variance>Y2, no calibration is performed on the allocation process.

7. A data processing device, which operates according to a data processing method according to any one of claims 1 to 6, characterized in that: include: The data package confirmation end confirms the data package that needs to be processed at the current stage. The data package is input by relevant personnel or data is collected from different ports that are set. A group of data packages is confirmed every group of collection cycles. The computing power equalization processing end distinguishes different types of data in the data package, confirms the total amount of features associated with the different types of data from the distinguished different types of data, confirms the total amount of binary associated with the corresponding type of data based on the confirmed total amount of features and the associated binary features, and performs equalization of conversion computing power based on the confirmed total amount of binary of the different types of data to determine the equalized computing power associated with the corresponding type of data; The real-time control end of the conversion process executes the same number of conversion threads based on the total number of different types of data in the total data package. Each conversion thread performs binary conversion on a group of type data to obtain binary conversion data of the corresponding type data, and the execution computing power of the conversion thread associated with the corresponding type data is the confirmed average computing power. The actual conversion amount and conversion rate of each thread are monitored in real time, and the real-time monitored data are analyzed in real time to readjust the average computing power associated with each thread. The data reading and writing synchronization end reads and writes the total data package after the data conversion is completed. From the historical data of different binary conversion data, the different reading and writing rates associated with different CPU usage ratio values ​​are confirmed, and then based on the specific usage ratio that the current CPU can use, this specific usage ratio is evenly distributed to the reading and writing processes of different binary conversion data, so that multiple different binary conversion data can synchronously complete the reading, writing and storage processes within the standard time difference.

8. A storage system for storing data generated by a data processing method according to any one of claims 1 to 6, characterized in that: The data packages that complete the data reading, writing and storage processes at different stages are stored, and different binary conversion data in the data packages are stored in different storage partitions.

9. A computer-readable medium, the computer-readable medium being used to provide preset data for a data processing method according to any one of claims 1 to 6, characterized in that: Multiple sets of preset values ​​are stored internally, including preset value Y1, which is used to compare the confirmed total processing characteristics CT; A preset value Y2 is also included for comparing the confirmed calibration variance.

Images