Data processing method capable of saving storage and computing power and related equipment

Abstract

The invention relates to a data processing method capable of saving storage and computing power and related equipment. Comprising the following steps: constructing a data processor and a corresponding monitoring scheduler to monitor to-be-processed data; performing data preprocessing through a data processor to obtain initial data; the initial data is split through a data processor based on a preset algorithm and the internal relevance of the initial data, a plurality of data subsets are obtained, and each data subset comprises part of features of the initial data; the data processor compresses all the data subsets based on a first preset compression algorithm to obtain corresponding data blocks, and compresses all the data blocks based on a second preset compression algorithm to obtain a plurality of target data components; and constructing a resource scheduler to allocate the target data component to the corresponding system processor for processing according to the data request of the data processor. By implementing the invention, the resource consumption in the data system processor can be effectively reduced, and the demand on the data system processor is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a data processing method and related equipment for saving storage and computing power. Background Art

[0002] In traditional data processing technology, data usually needs to be processed under the active scheduling of system processors (such as CPU, GPU, etc.). With the explosive growth of data volume, traditional data processing technology faces severe challenges, especially in terms of resource consumption and processing efficiency. In traditional methods, data system processors (such as CPU, GPU, etc.) need to actively schedule data. This makes the data processing process require more resources from the data system processor, making the entire data processing process not only resource-intensive but also inefficient. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a data processing method and related equipment that saves storage and computing power in response to the above-mentioned technical defects of the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is: constructing a data processing method that saves storage and computing power, including: Constructing a data processor and a monitoring scheduler corresponding to the data processor, so as to monitor the data to be processed through the monitoring scheduler; Preprocessing the data to be processed by the data processor to obtain preprocessed initial data; Splitting the initial data by the data processor based on a preset algorithm and according to the inherent relevance of the initial data to obtain a plurality of data subsets, wherein each data subset contains part of the features of the initial data; The data processor compresses all data subsets based on a first preset compression algorithm to obtain corresponding data blocks, and compresses all data blocks based on a second preset compression algorithm to obtain a plurality of target data components; A resource scheduler corresponding to the data processor is constructed to allocate the target data component to the corresponding system processor for processing according to the data request of the data processor.

[0005] Preferably, in the data processing method of the present invention, compressing the data block based on a second preset compression algorithm to obtain a plurality of target data components includes: Construct a convolutional network consisting of the first convolutional layer, the second convolutional layer, and the third convolutional layer; Setting a convolution kernel size and a step size of the first convolution layer to convert the data block into a data block combination through the first convolution layer; Setting an activation function and a convolution kernel of the second convolution layer to convert the data block combination into data components through the second convolution layer; The dimension parameter of the third convolutional layer is set to convert the data component into a high-dimensional data component through the third convolutional layer, and use the high-dimensional data component as the target data component.

[0006] Preferably, in the data processing method of the present invention, the preprocessing of the data to be processed to obtain preprocessed initial data includes: The data to be processed is cleaned and standardized to remove useless information and noise and obtain the initial data.

[0007] Preferably, in the data processing method of the present invention, splitting the initial data based on a preset algorithm and according to the internal correlation of the initial data includes: The initial data is intelligently split using the DataGenome algorithm.

[0008] Preferably, in the data processing method of the present invention, compressing all data subsets respectively based on the first preset compression algorithm to obtain corresponding data blocks includes: All data subsets are compressed separately using the LZ77 algorithm to obtain corresponding data blocks.

[0009] Preferably, in the data processing method of the present invention, the method further comprises: generating a static resource allocation table corresponding to the system processor to record processing tasks executable by the system processor; The resource scheduler allocates the target data component to the corresponding system processor according to the static resource allocation table.

[0010] Preferably, in the data processing method of the present invention, the method further comprises: The resource scheduler allocates the target data components to corresponding system processors according to the priorities of the target data components.

[0011] The present invention also constructs an electronic device, which includes: a module for executing the method described above.

[0012] The present invention also constructs an electronic device, comprising a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method described above.

[0013] The present invention also constructs a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.

[0014] The implementation of a data processing method and related equipment for saving storage and computing power of the present invention has the following beneficial effects: through the data processing process, the resource consumption in the system processor can be effectively reduced, and the demand for the system processor can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which: Figure 1 It is a program flow chart of an embodiment of a data processing method for saving storage and computing power of the present invention; Figure 2 is a program flow chart of another embodiment of a data processing method for saving storage and computing power of the present invention; Figure 3 It is a program flow chart of another embodiment of a data processing method for saving storage and computing power of the present invention. DETAILED DESCRIPTION

[0016] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0017] like Figure 1 , shows an embodiment of a data processing method of the present invention that saves storage and computing power. Figure 1 In one embodiment of a data processing method of the present invention shown, the following processes are included: S1, constructing a data processor and a monitoring scheduler corresponding to the data processor to monitor the data to be processed through the monitoring scheduler; S2, preprocessing the data to be processed by the data processor to obtain preprocessed initial data; S3, splitting the initial data by the data processor based on a preset algorithm and according to the intrinsic correlation of the initial data to obtain a number of data subsets, wherein each data subset contains part of the characteristics of the initial data; S4, compressing all data subsets by the data processor based on a first preset compression algorithm to obtain corresponding data blocks, and compressing all data blocks based on a second preset compression algorithm to obtain a number of target data components; S5, constructing a resource scheduler corresponding to the data processor to allocate the target data components to the corresponding system processor for processing according to the data request of the data processor.

[0018] Based on step S1, a corresponding data processor and a monitoring scheduler are constructed for the data to be processed each time, and the data to be processed is monitored by the monitoring scheduler. Among them, the monitoring action of the data to be processed is only used to obtain the data status of the data to be processed, that is, for example, to obtain what type of data the data to be processed is, to obtain the start or end time point of the data to be processed, to obtain the size of the data to be processed, etc. Here, the monitoring action of the data to be processed does not obtain the specific data content of the data to be processed, that is, it does not involve the leakage or disclosure of the specific content of the data. After the data processor obtains the relevant information of the data to be processed through the monitoring scheduler, it performs the next step as required.

[0019] Based on step S2, the data processor pre-processes the acquired data to be processed to obtain initial data corresponding to the data to be processed.

[0020] In one embodiment, the preprocessing of the data to be processed to obtain the preprocessed initial data includes: cleaning and standardizing the data to be processed to remove useless information and noise therein and obtain the initial data. That is, based on the preprocessing process in step S2, it can include cleaning and standardizing the data to be processed, and its specific purpose is to remove useless information and noise therein. Among them, the cleaning and standardization of the data can adopt some currently existing data denoising processes. Its specific process can be related to the type of the processed data, for example, audio data or video data, pure text data can be denoised in their own most suitable way.

[0021] Based on step S3, the data processor splits the obtained initial data through a preset algorithm. In the specific splitting process, the inherent correlation of each data in the initial data is taken into account, so that each of the several data subsets obtained after the splitting contains some features of the initial data. Through this process, it can be understood that because each data subset has a specific relationship with the features of the initial data, the correlation relationship between the data subset and the initial data can be obtained. Conversely, after combining several data subsets, all the features of the initial data can be included. If several data subsets are combined, all the features of the initial data can be completely restored to obtain complete initial data.

[0022] Based on step S4, all the obtained data subsets are compressed by the first preset compression algorithm to obtain a number of data blocks. It can be understood that each data subset corresponds to one or more data blocks. That is, the number of compressed data blocks can be obtained according to the compression process or the amount of data in the data subset. Then the data blocks corresponding to each data subset are further compressed to obtain the final target data components corresponding to each data subset. It can be understood that each target data component has a corresponding relationship with each data subset. Because each data subset contains some features of the data to be processed, the obtained target data component also contains some features of the data to be processed, so the data content of the data to be processed can be obtained based on the target data component.

[0023] Based on step S5, a corresponding resource scheduler is constructed for each data processor, and resources are allocated to the obtained target data components through the resource scheduler, so that each target data component is allocated to the corresponding system processor for processing. That is, the system processor is scheduled to realize the processing of each target data component, and then the final processing process of the entire data to be processed is realized. Among them, when the resource scheduler performs system processor scheduling, it is necessary to consider the data processing requirements of the data processor for each data component and perform a specific system processor scheduling process.

[0024] It can be understood that the above system processor is the CPU or GPU unit that actually performs data processing in the system, that is, the functional component that actually performs data processing in the system. Through the above process, the ultimate goal is to achieve that the data to be processed can be reasonably processed by the system processor and optimize the data processing process of the entire system.

[0025] In one embodiment, if Figure 2 As shown, in step S4, the data block is compressed based on the second preset compression algorithm to obtain a number of target data components, which specifically includes the following steps: S41, constructing a convolutional network including a first convolutional layer, a second convolutional layer and a third convolutional layer; S42, setting the convolution kernel size and step size of the first convolutional layer to convert the data block into a data block combination through the first convolutional layer; S43, setting the activation function and convolution kernel of the second convolutional layer to convert the data block combination into a data component through the second convolutional layer; S44, setting the dimension parameter of the third convolutional layer to convert the data component into a high-dimensional data component through the third convolutional layer, and taking the high-dimensional data component as the target data component.

[0026] Based on step S31, a multi-layer convolutional network is constructed, wherein the multi-layer convolutional network includes a first convolutional layer, a second convolutional layer, and a third convolutional layer. That is, through CNN and convolution algorithms, GPU computing power is used to accelerate multi-layer convolutional calculation tasks, and data is gradually transformed from a simple structure to a more complex structure. Among them, each convolutional layer is performed under the acceleration of the GPU, ensuring processing speed and efficiency. Through this multi-layer convolutional calculation, the data is gradually refined and optimized, providing high-quality input for the final data application.

[0027] Based on step S32, after the data block is input to the first convolutional layer through the input end of the convolutional network, the data block can be converted into a data block combination through the first convolutional layer. Among them, the specific process of converting the data block into the data block combination can be optimized by changing the convolution kernel size and step size in the first convolutional layer. For example, during the compression process of the data block, if two data bytes need to be combined, the step size can be set to 2, and if 4 data bytes need to be combined and compressed, the step size can be set to 4. Here, the data byte is understood as a unit of measurement for data. By setting the step size, it is possible to select how many units of data are compressed during the compression process in the data block. It can be understood that the compression process of the first convolutional layer is based on the data blocks corresponding to each data subset, so that the data of the data subset is converted into a data combination with a larger granularity.

[0028] Based on step S33, the output of the first convolution layer is used as the input of the second convolution layer, and the activation function and convolution kernel of the second convolution layer are set so that the data block combination is converted into a data component through the second convolution layer. The selection of the activation function and the convolution kernel can be selected according to the data type of the data to be processed.

[0029] Based on step S34, by setting the dimension parameters of the third convolutional layer, the data component output by the second convolutional layer passes through the third convolutional layer, and outputs a high-dimensional data component, that is, the final target data component is obtained.

[0030] In a specific embodiment, when the data to be processed is a medical document, the first convolution layer is processed using a general word compression algorithm, the second convolution layer is processed using a commonly used drug name compression algorithm, and the third convolution layer is processed using a medical principle professional terminology compression algorithm.

[0031] In one embodiment, based on step S3, the initial data is split based on a preset algorithm and according to the intrinsic correlation of the initial data, including: intelligently splitting the initial data through the DataGenome algorithm. Specifically, when the initial data is split, the DataGenome algorithm can be used to perform a specific splitting process. As a network inference algorithm for gene expression data, the DataGenome algorithm inherits some genes or some features of the initial data during the data splitting process, so that the split data is associated with the initial data.

[0032] In one embodiment, based on step S4, compressing all data subsets based on the first preset compression algorithm to obtain corresponding data blocks includes: compressing all data subsets respectively using the LZ77 algorithm to obtain corresponding data blocks. Specifically, compressing data using the LZ77 algorithm can greatly reduce data storage space and transmission costs.

[0033] like Figure 3 As shown, in one embodiment, in the data processing method of the present invention, after constructing the resource scheduler, the following steps are further performed: S6, generating a static resource allocation table corresponding to the system processor to record the processing tasks that can be executed by the system processor; S7, the resource scheduler allocates the target data component to the corresponding system processor according to the static resource allocation table. Specifically, a static resource allocation table is allocated to each system processor, and the static resource allocation table records the data type and processing capability that the system processor can process. When the data is delivered through the resource scheduler, the system processor directly processes the data according to the static resource allocation table without the need for additional resource scheduling. Through this process, multiple system processors can simultaneously receive different target data components for processing. Due to the static configuration of the system processor, the system can achieve efficient parallel data processing without increasing additional resource consumption.

[0034] Furthermore, in the data processing method of the present invention, it also includes: the resource scheduler allocates the target data component to the corresponding system processor according to the priority of the target data component. That is, when the resource scheduler allocates the target data component, it also needs to consider the priority of the target data component, that is, when a system processor can process multiple target data components, it can give priority to the target data component with a higher priority among the multiple target data components.

[0035] In the present invention, based on the active delivery of data and the static processing of the system processor in the above process, resource consumption is greatly reduced, and the efficiency and speed of data processing are improved. That is, the whole process converts conventional data (corresponding to hot data) into data that is compressed into high-dimensional data processing according to the definition of high-dimensional data, that is, cold data, through efficient data splitting and compression. It is suitable for data processing in a big data environment, especially for application scenarios of cloud computing and data centers. Moreover, the method of the present invention can also process more data simultaneously through multi-threading of the system processor, further improving the efficiency of data processing.

[0036] In addition, an electronic device of the present invention has the function of implementing the corresponding steps performed in the above method. Each function can be implemented by hardware, or by hardware executing corresponding software. The corresponding hardware or software includes one or more modules corresponding to the above functions. That is, the steps in the above method are respectively executed by one or more modules. The specific coordination operations between the modules can refer to the specific process of the above method, which will not be repeated here. In addition, an electronic device of the present invention may also include a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program to implement any of the above methods. Specifically, according to an embodiment of the present invention, the process described with reference to the flowchart above can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed by an electronic device and when executed, it performs the above functions defined in the method of the embodiment of the present invention. The electronic device in the present invention can be a terminal such as a notebook, a desktop, a tablet computer, a smart phone, or a server.

[0037] In addition, a computer storage medium of the present invention stores a computer program thereon, and when the computer program is executed by a processor, any of the above methods is implemented. Specifically, it should be noted that the above-mentioned computer-readable medium of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0038] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0039] It can be understood that the above embodiments only express the preferred implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the patent scope of the present invention. It should be pointed out that, for ordinary technicians in this field, the above technical features can be freely combined without departing from the concept of the present invention, and several deformations and improvements can be made, which all belong to the protection scope of the present invention. Therefore, all equivalent changes and modifications made to the scope of the claims of the present invention should belong to the coverage of the claims of the present invention.

Claims

1. A data processing method for saving storage and computing power, characterized in that: include: Constructing a data processor and a monitoring scheduler corresponding to the data processor, so as to monitor the data to be processed through the monitoring scheduler; Preprocessing the data to be processed by the data processor to obtain preprocessed initial data; Splitting the initial data by the data processor based on a preset algorithm and according to the inherent relevance of the initial data to obtain a plurality of data subsets, wherein each data subset contains part of the features of the initial data; The data processor compresses all data subsets based on a first preset compression algorithm to obtain corresponding data blocks, and compresses all data blocks based on a second preset compression algorithm to obtain a plurality of target data components; A resource scheduler corresponding to the data processor is constructed to allocate the target data component to the corresponding system processor for processing according to the data request of the data processor.

2. The data processing method according to claim 1, characterized in that: The compressing the data block based on the second preset compression algorithm to obtain a plurality of target data components includes: Construct a convolutional network consisting of the first convolutional layer, the second convolutional layer, and the third convolutional layer; Setting a convolution kernel size and a step size of the first convolution layer to convert the data block into a data block combination through the first convolution layer; Setting an activation function and a convolution kernel of the second convolution layer to convert the data block combination into data components through the second convolution layer; The dimension parameter of the third convolutional layer is set to convert the data component into a high-dimensional data component through the third convolutional layer, and use the high-dimensional data component as the target data component.

3. The data processing method according to claim 1, characterized in that: The preprocessing of the data to be processed to obtain preprocessed initial data includes: The data to be processed is cleaned and standardized to remove useless information and noise and obtain the initial data.

4. The data processing method according to claim 1, characterized in that: The splitting of the initial data based on a preset algorithm and according to the internal relevance of the initial data includes: The initial data is intelligently split using the DataGenome algorithm.

5. The data processing method according to claim 1, characterized in that: The step of compressing all data subsets based on a first preset compression algorithm to obtain corresponding data blocks includes: All data subsets are compressed separately using the LZ77 algorithm to obtain corresponding data blocks.

6. The data processing method according to claim 1, characterized in that: The method further comprises: generating a static resource allocation table corresponding to the system processor to record processing tasks executable by the system processor; The resource scheduler allocates the target data component to the corresponding system processor according to the static resource allocation table.

7. The data processing method according to claim 6, characterized in that: The method further comprises: The resource scheduler allocates the target data components to corresponding system processors according to the priorities of the target data components.

8. An electronic device, characterized in that: The electronic device comprises: a module for executing the method according to any one of claims 1 to 7.

9. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images