Data processing method, device, computer equipment and storage medium

Abstract

The present application relates to a data processing method, device, computer equipment, storage medium and computer program product. The method comprises: receiving a data stream through a data access unit, wherein the data stream contains a compressed data packet. Based on a preset data storage path, the data stream is stored in a distributed storage space. According to a preset time interval and a preset data storage path, the data in the stored data stream is scanned to detect whether there is a signal file in the data stream. If a signal file exists, the compressed data packet in the data stream is decompressed, and the decompressed data is stored in the distributed storage space based on a preset data query path. The use of this method improves the efficiency of reading and writing data.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the rapid development and widespread application of computer network and database technology, information management in all walks of life has entered a new era.

[0003] The current database system decompresses the compressed package of the report file and stores it in a centralized database in the form of large fields. When a user performs a data query, all the data contents in the centralized database are traversed to obtain the target report file.

[0004] However, the centralized database has a single storage location and limited storage space. When the data access volume to the same storage location is large, the data read and write performance is poor. Moreover, when the data volume of the stored report files gradually increases, the storage capacity of the centralized database cannot meet the growing demand for report data. Summary of the invention

[0005] Based on this, it is necessary to provide a data processing method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical issues.

[0006] In a first aspect, the present application provides a data processing method. The method comprises:

[0007] receiving a data stream, wherein the data stream includes compressed data packets;

[0008] Based on a preset data storage path, storing the data stream in a first distributed storage space;

[0009] Scanning the data in the stored data stream according to the preset time interval and the preset data storage path to detect whether there is a signal file in the data stream;

[0010] If the signal file exists, the compressed data packets in the data stream are decompressed, and the decompressed data are stored in the second distributed storage space based on a preset data query path, so that data query is performed in the second distributed storage space.

[0011] In one embodiment, if the signal file exists, decompressing the compressed data packets in the data stream comprises:

[0012] If the signal file exists, in response to the triggering of the signal file, each compressed data packet is split according to a preset data block size to obtain a plurality of data blocks to be decompressed;

[0013] Splitting each of the data blocks to be decompressed according to a preset data slice size to obtain a plurality of data slices to be decompressed;

[0014] By calling the data processing thread, the compressed data in each of the data slices is parsed and calculated to obtain the decompressed slice data;

[0015] The decompressed segmented data are aggregated to obtain a decompressed report file.

[0016] In one embodiment, storing the decompressed data in the second distributed storage space based on a preset data query path includes:

[0017] Matching a corresponding data query path for each decompressed report file data, and storing the decompressed report file data in a target storage space of the distributed storage space based on each data query path;

[0018] Extracting characteristic information of each of the decompressed report file data and generating a data identifier of the report file data;

[0019] A data query list including the correspondence between the data identifier and the data query path is established, and the data query list is pushed to the front end for data query.

[0020] In one embodiment, the method further comprises:

[0021] receiving a data query request, wherein the data query request carries a target data identifier;

[0022] According to the target data identifier, determine the target data query path corresponding to the target data identifier in the correspondence between each data identifier and the data query path included in the data query list;

[0023] Based on the target data query path, the target report file data is obtained in the second distributed storage space and fed back to the front end in the form of a byte stream.

[0024] In a second aspect, the present application further provides a data processing device. The device comprises:

[0025] A receiving module, used for receiving a data stream, wherein the data stream includes a compressed data packet;

[0026] A storage module, configured to store the data stream in a first distributed storage space based on a preset data storage path;

[0027] A detection module, used to scan the data in the stored data stream according to a preset time interval and the preset data storage path, and detect whether there is a signal file in the data stream;

[0028] A processing module is used to decompress the compressed data packets in the data stream if the signal file exists, and store the decompressed data in the second distributed storage space based on a preset data query path, so as to perform data query in the second distributed storage space.

[0029] In a third aspect, the present application further provides a data processing system, the data processing system comprising:

[0030] A data access unit, configured to receive a data stream, wherein the data stream includes a compressed data packet;

[0031] A data storage unit, configured to store the data stream in a first distributed storage space based on a preset data storage path;

[0032] A data processing unit is used to scan the data in the stored data stream according to a preset time interval and the preset data storage path, and detect whether there is a signal file in the data stream; if the signal file exists, decompress the compressed data packet in the data stream, and store the decompressed data in the second distributed storage space based on the preset data query path.

[0033] In one embodiment, the data processing system further includes:

[0034] The report data application unit is used to receive a data query request, and determine, according to the target data identifier carried in the data query request, a target query path corresponding to the target data identifier in the correspondence between each data identifier information and the data query path contained in the data query list;

[0035] Based on the target query path, the target report file data is obtained in the second distributed storage space and fed back to the front end in the form of a byte stream.

[0036] In one of the embodiments, the data processing unit includes a scheduling module and a resource allocation module;

[0037] The scheduling module is used to scan the data in the stored data stream according to the preset time interval and the preset data storage path, and detect whether there is a signal file in the data stream;

[0038] The resource allocation module is used to allocate data processing threads. If the signal file exists, the compressed data packets in the data stream are decompressed by the data processing threads, and the decompressed data are stored in the second distributed storage space based on a preset data query path.

[0039] In a fourth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0040] receiving a data stream, wherein the data stream includes compressed data packets;

[0041] Based on a preset data storage path, storing the data stream in a first distributed storage space;

[0042] Scanning the data in the stored data stream according to the preset time interval and the preset data storage path to detect whether there is a signal file in the data stream;

[0043] If the signal file exists, the compressed data packets in the data stream are decompressed, and the decompressed data are stored in the second distributed storage space based on a preset data query path, so that data query is performed in the second distributed storage space.

[0044] In a fifth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0045] receiving a data stream, wherein the data stream includes compressed data packets;

[0046] Based on a preset data storage path, storing the data stream in a first distributed storage space;

[0047] Scanning the data in the stored data stream according to the preset time interval and the preset data storage path to detect whether there is a signal file in the data stream;

[0048] If the signal file exists, the compressed data packets in the data stream are decompressed, and the decompressed data are stored in the second distributed storage space based on a preset data query path, so that data query is performed in the second distributed storage space.

[0049] In a sixth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0050] receiving a data stream, wherein the data stream includes compressed data packets;

[0051] Based on a preset data storage path, storing the data stream in a first distributed storage space;

[0052] Scanning the data in the stored data stream according to the preset time interval and the preset data storage path to detect whether there is a signal file in the data stream;

[0053] If the signal file exists, the compressed data packets in the data stream are decompressed, and the decompressed data are stored in the second distributed storage space based on a preset data query path, so that data query is performed in the second distributed storage space.

[0054] The above-mentioned data processing method, device, computer equipment, storage medium and computer program product receive a data stream through a data access unit, wherein the data stream contains a compressed data packet. Then, the data processing unit stores the data stream in a distributed storage space based on a preset data storage path; scans the data in the stored data stream according to a preset time interval and the preset data storage path, and detects whether there is a signal file in the data stream. If the signal file exists, the compressed data packet in the data stream is decompressed, and the decompressed data is stored in the distributed storage space based on a preset data query path. Using this method, the compressed data packet in the data stream is stored in the distributed storage space under the preset data storage path. At the same time, the compressed data packet is decompressed based on the trigger of the signal file, and the decompressed data is stored in the corresponding data query path, so that the access and query of the data are decoupled, the reading and writing efficiency of the data is improved, and the distributed storage space is used to store the data, and the data storage capacity can be increased by continuously accessing new storage nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 An application environment diagram of a data processing method in an embodiment;

[0056] Figure 2 is a flow chart of a data processing method in one embodiment;

[0057] Figure 3 A schematic diagram of a flow chart of a data splitting process step in an embodiment;

[0058] Figure 4 A schematic diagram of a process for constructing a data query list in one embodiment;

[0059] Figure 5 A schematic diagram of a flow chart of a data query step in an embodiment;

[0060] Figure 6 A schematic diagram of a flow chart of a data storage step in an embodiment;

[0061] Figure 7 A flowchart of an example of a data query method in an embodiment;

[0062] Figure 8 is a structural block diagram of a data processing device in one embodiment;

[0063] Fig. 9 is a structural diagram of a data processing system in one embodiment;

[0064] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] The data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers, which is not limited in the embodiment of the present application. In the server 104, a data stream is received. Then, the server 104 stores the data stream in a first distributed storage space based on a preset data storage path. At the same time, the data processing unit in the server 104 scans the stored data stream according to a preset time interval and a preset data storage path, and detects whether there is a signal file in the data stream. If there is a signal file, the data processing unit decompresses the compressed data packet in the data stream, and stores the decompressed data in the second distributed storage space based on a preset data query path, so that data query can be performed based on the data query path. By adopting this data processing method, data access and query are decoupled, and the reading and writing efficiency of data is improved.

[0067] Optionally, the terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as a server cluster consisting of multiple servers.

[0068] In one embodiment, Figure 2 As shown, a data processing method is provided, which is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0069] Step 202: Receive a data stream, wherein the data stream includes compressed data packets.

[0070] In implementation, the server calls a preset API (Application Programming Interface) interface through a data access unit to receive a data stream transmitted by an upstream source system. Then, the data access unit transmits the received data stream to a data storage unit (also called a distributed storage system) for data storage.

[0071] Among them, the data stream may include compressed data packets in a compressed format, and may also include data in other file formats. At the same time, the data content in the data stream may be report data or data of other contents, which is not limited in the embodiments of the present application.

[0072] Step 204: store the data stream into the first distributed storage space based on a preset data storage path.

[0073] In implementation, based on the data storage information provided in advance by the upstream data provider, the server formulates a data storage path, wherein the addresses pointed to in the data storage path are divided into levels and recorded in the format of a data storage directory. Then, the data stream is transmitted based on the data storage directory and transmitted to the designated storage space (i.e., the first distributed storage space) of the data storage unit for storage.

[0074] Step 206 , scanning the data in the stored data stream according to the preset time interval and the preset data storage path, and detecting whether there is a signal file in the data stream.

[0075] In implementation, the data stream transmitted by the upstream data provider includes not only compressed data packets but also signal files for triggering the data processing process. The data processing unit can start processing the data in the data stream based on the triggering of the signal file. Specifically, along with the real-time access of the data stream, the data processing unit of the server periodically scans the data in the target distributed storage space (i.e., the first distributed storage space) pointed to by the data storage directory according to the preset time interval and the data storage path represented by the data storage target, and detects whether there is a signal file for the data already stored in the distributed storage space.

[0076] Among them, the signal file is a program code containing a trigger signal that can trigger the data processing process. In addition, the file format of the signal file is based on the pre-definition between the upstream data provider and the data receiver. Therefore, the embodiment of the present application does not limit the file format of the signal file.

[0077] Step 208: If the signal file exists, decompress the compressed data packet in the data stream, and store the decompressed data in the second distributed storage space based on a preset data query path.

[0078] In implementation, if the data processing unit in the server detects that there is a signal file in the data of the stored data stream, based on the trigger of the signal file, the data processing unit decompresses the compressed data packets contained in the data stream, and based on the preset data query path, stores the decompressed data in the second distributed storage space pointed to by the data query path. The second distributed storage space and the first distributed storage space belong to the same distributed storage system, but the storage locations represented by the two storage spaces are different.

[0079] Specifically, if the data contained in the compressed data packet is report data, the report data to be decompressed belonging to the same business report is decompressed by the data processing unit to obtain a corresponding report file. Then, the data processing unit matches the corresponding data query path for the report file, and stores the report file in the target storage area of ​​the distributed storage space (i.e., the second distributed storage space) based on the data query path, so that when the user performs data query through the terminal device, he can directly perform data query through the matching data query path.

[0080] In the above data processing method, a data stream is received by a data access unit, and a data storage unit stores the data stream in a distributed storage space based on a preset data storage path. Then, the data processing unit scans the data in the stored data stream according to a preset time interval and a preset data storage path to detect whether there is a signal file in the data stream. If there is a signal file, the compressed data packet contained in the data stream is decompressed, and the decompressed data is stored again in the distributed storage space of the data storage unit based on a preset data query path. Using this method, the compressed data packet in the data stream is stored in the distributed storage space under the preset data storage path. At the same time, the compressed data packet is decompressed based on the trigger of the signal file, and the decompressed data is stored in the corresponding data query path, so that the access and query of the data are decoupled, the reading and writing efficiency of the data is improved, and the distributed storage space is used to store the data, and the data storage capacity can be increased by continuously accessing new storage nodes.

[0081] In one embodiment, Figure 3As shown, since the size of the compressed data packet in the data stream is not limited in the present application, it is necessary to split the larger compressed data packet before triggering the data processing process. Therefore, if the signal file exists in step 208, the specific processing process of decompressing the compressed data packet in the data stream includes:

[0082] Step 302: If a signal file exists, in response to the triggering of the signal file, each compressed data packet is split according to a preset data block size to obtain a plurality of data blocks to be decompressed.

[0083] In implementation, if the data processing unit detects the presence of a signal file in the data stream, in response to the triggering of the signal file, the data processing unit splits larger (ie, the size of the data packet is larger than a preset threshold) compressed data packets in the data stream.

[0084] Specifically, since the traditional relational database uses file storage for storage, there are restrictions on the size of a single file, and larger files cannot be stored. However, the present application uses a distributed data packet and data block storage method, which does not limit the size of the compressed data packet, thereby improving the convenience of data storage. Furthermore, there may be larger compressed data packets in the distributed storage space, so before applying the data, the data processing unit needs to split the larger compressed data packets.

[0085] The data processing unit is pre-equipped with a splitting unit, i.e., a data block with M bytes as the standard, and the splitting unit is used as the compressed data packet splitting unit to split a larger compressed data packet, the purpose of which is to obtain multiple block data blocks for subsequent processing. For example, the byte size of the preset data splitting unit is M equal to 10, then when a compressed data packet larger than the splitting unit appears in the data stream (for example, the size of the current compressed data packet is 100 bytes), the size of the splitting unit (10 bytes) is used as a unit, i.e., 100÷10=10 (pieces), then the 100-byte (MB) compressed data packet is split to obtain 10 10-byte compressed data blocks.

[0086] Step 304: split each data block to be decompressed according to a preset data slice size to obtain a plurality of data slices to be decompressed.

[0087] In implementation, the data in the data block can be further split and processed according to the data features contained in the compressed data packet. For example, when the data contained in each data block is report data, each data block is split into data slices according to the size standard of a single report data to obtain multiple data slices to be decompressed.

[0088] Step 306, by calling the data processing thread, the compressed data in each data slice is parsed and calculated to obtain decompressed slice data.

[0089] During implementation, in order to improve data processing efficiency, for the multiple data slices to be decompressed obtained by splitting, the resource allocation module in the data processing unit calls multiple data processing threads for parallel processing based on the data volume of the obtained data slices, parses and calculates the compressed data in each data slice, and obtains the decompressed slice data.

[0090] Step 308: aggregate the decompressed segmented data to obtain a decompressed report file.

[0091] In implementation, the data processing unit aggregates the decompressed fragmented data belonging to the same report file to obtain a decompressed report file.

[0092] In this embodiment, the size of the compressed data packet in the accessed data stream is not displayed through the storage method of data blocks. Before the data is applied, the compressed data packet exceeding the preset byte size is split and processed, and the corresponding data processing threads are allocated to parallelly process the multiple data fragments to be decompressed obtained by splitting the compressed data packet, so that the access storage of the data stream and the data application in the data stream are decoupled, thereby improving the data reading and writing efficiency.

[0093] In one embodiment, Figure 4 As shown, after decompressing the data in the compressed data packet, the data processing unit stores the decompressed data in other storage spaces of the data storage unit except the data storage space stored when the data stream is accessed. The specific processing steps of storing the decompressed data in the distributed storage space based on the preset data query path in step 208 include:

[0094] Step 402 : Match the corresponding data query path for each decompressed report file data, and store the decompressed report file data into the target storage space of the distributed storage space based on each data query path.

[0095] In implementation, the data processing unit in the server matches the corresponding data query path for each decompressed report file data. For example, when 10 report file data are decompressed from a compressed data packet, the corresponding data query path is matched for each report file data, wherein the data query path can be hierarchically divided into corresponding storage spaces in the form of a directory. Then, based on each matched data query path, the data processing unit stores the corresponding decompressed report file data in the target storage space of the distributed storage space. The target storage space is the data storage space pointed to by the data query path.

[0096] Step 404: extract characteristic information of each decompressed report file data and generate a data identifier of the report file data.

[0097] During implementation, the data processing unit extracts characteristic information of each decompressed report file data and generates a data identifier of the report file data. Specifically, the data processing unit can extract the name information of the report header of each report file data as the characteristic information of the report file data, and then generate a data identifier of the corresponding report file data based on the name information of the report file. For example, if the name information of the report file data is "XX Report", the report file data is uniquely identified by the name information.

[0098] Step 406: Create a data query list containing the correspondence between the data identifier and the data query path, and push the data query list to the front end for data query.

[0099] In implementation, the data processing unit establishes a data query list of the corresponding relationship between the data identifier and the data query path, and pushes the data query list to the front end of the report application unit to provide it to the user for data query application.

[0100] In addition, the data query list includes report file data 1 corresponding to data query path 1, report file data 2 corresponding to data query path 2, report file data 3 corresponding to data query path 3... Among them, the data query paths corresponding to the report file data may involve overlapping parts. For example, the storage space pointed to in data query path 1 corresponds to hard disk 1 of node 1 in the distributed storage system, and the storage space pointed to in data query path 2 also corresponds to hard disk 1 of node 1 in the distributed storage system. However, data query path 1 and data query path 2 point to different data storage areas in hard disk 1. Therefore, data query path 1 and data query path 2 are still two different data query paths. Users can uniquely query target message data based on different data query paths.

[0101] Specifically, the server backend system establishes a correspondence between the data identifier and the data query path for each report file data and the matching data query path, generates a data query list, pushes the data query list to the front end, and outputs and displays it in the form of a list of options for each report file, so that users can perform data query and application based on the report file option list displayed on the front end.

[0102] Optionally, in the distributed storage space of the data storage unit (also called a distributed storage system), in order to ensure high availability of data, the number of copies of the distributed storage can also be set. At the same time, in order to meet the growing demand for storage capacity, the distributed storage space can achieve horizontal expansion of storage capacity by connecting to new storage nodes, and relying on the diversity of distributed storage interfaces, it can make data application scenarios diverse and provide data application convenience.

[0103] In one embodiment, Figure 5 As shown, in the present application, the server processes the stored compressed data packets according to a preset time period after storing the data for the accessed source data stream to obtain decompressed data, and then can apply for data query and application to the decompressed data. Specifically, the method also includes:

[0104] Step 502: Receive a data query request, wherein the data query request carries a target data identifier.

[0105] In the implementation, the user performs a target data query based on the data query window provided by the front end in the report application unit, and inputs a data query request through the front end, which carries the target data identifier. The front end transmits the data query request to the server backend system (also called the backend service system). The report application unit in the server backend system is built based on SpringBoot, receives the data query request from the front end, and queries and obtains the target report file data.

[0106] Step 504: According to the target data identifier, determine the target data query path corresponding to the target data identifier in the correspondence between each data identifier and the data query path included in the data query list.

[0107] In implementation, the back-end system of the server (also called the back-end service system) identifies the target data identifier carried therein based on the trigger of the received data query request, and determines the target data query path corresponding to the target data identifier in the correspondence between the data identifiers and the data query paths contained in the pre-built data query list.

[0108] Step 506: based on the target data query path, obtain the target report file data in the distributed storage space.

[0109] In the implementation, the backend system of the server calls the corresponding data transmission interface and obtains the target report file data in the target data storage space pointed to by the target data query path. Then, the backend system of the server pushes the obtained target report file data to the front end in the form of byte stream for display, so that users can browse and view it, avoiding the secondary landing of the report file data. At the same time, the report application unit can be deployed in a containerized manner to support load balancing and horizontal expansion.

[0110] Optionally, in addition to recording the correspondence between the data identifier (data feature attribute information (description information)) and the data query path of each report file data in the data stream, the data query list in the data storage unit may further record the data query frequency of each report file data, and determine the query priority of the data query list based on the data query frequency, thereby improving the data report query efficiency.

[0111] In one embodiment, Figure 6 As shown, an example of a data processing method is provided, which specifically includes the following steps:

[0112] Step 601: A data access unit receives a data stream from a source end system and transmits data in the data stream to a data unit. The data stream includes a compressed data packet and a signal file.

[0113] Step 602: The data storage unit stores the data received in the data stream based on a preset data storage path.

[0114] Step 603, the data processing unit queries whether there is a signal file in the data storage unit according to the data storage path and the preset time period. If there is a signal file, the compressed data packet is read, the compressed data packet is decompressed, and the decompressed data is re-stored in the distributed storage space of the data storage unit via another data query path.

[0115] In one embodiment, Figure 7 As shown, an example of a data query method is provided, which specifically includes the following steps:

[0116] Step 701: The user sends a data query request to the report application unit of the backend service system through the front end of the report application unit. The data query request carries a target data identifier.

[0117] Step 702: The report application unit determines the correspondence between the target data identifier and the target data query path in the correspondence included in the data query list according to the target data identifier carried in the data query request.

[0118] Step 703: the report application unit obtains the target report file data in the target storage space of the data storage unit according to the determined target data query path, and pushes the target report file data to the front end in the form of byte stream for output display.

[0119] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0120] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0121] In one embodiment, Figure 8 As shown, a data processing device 800 is provided, including: a receiving module 801, a storage module 802, a detection module 803 and a processing module 804, wherein:

[0122] A receiving module 801 is used to receive a data stream, wherein the data stream includes a compressed data packet;

[0123] A storage module 802, configured to store the data stream in a first distributed storage space based on a preset data storage path;

[0124] A detection module 803 is used to scan the data in the stored data stream according to a preset time interval and a preset data storage path to detect whether there is a signal file in the data stream;

[0125] The processing module 804 is used to decompress the compressed data packets in the data stream if there is a signal file, and store the decompressed data in the second distributed storage space based on a preset data query path, so as to perform data query in the second distributed storage space.

[0126] In one embodiment, the processing module 804 is specifically configured to split each compressed data packet according to a preset data block size to obtain a plurality of data blocks to be decompressed in response to a trigger of the signal file if there is a signal file;

[0127] Splitting each data block to be decompressed according to a preset data fragment size to obtain multiple data fragments to be decompressed;

[0128] By calling the data processing thread, the compressed data in each data slice is parsed and calculated to obtain the decompressed slice data;

[0129] The decompressed segmented data is aggregated and processed to obtain a decompressed report file.

[0130] In one embodiment, the processing module 804 is further used to match a corresponding data query path for each decompressed report file data, and store the decompressed report file data into a target storage space of the distributed storage space based on each data query path;

[0131] Extract characteristic information of each decompressed report file data and generate data identifier of the report file data;

[0132] A data query list including the correspondence between data identifiers and data query paths is established, and the data query list is pushed to the front end for data query.

[0133] In one embodiment, the apparatus 800 further includes:

[0134] A receiving module, used for receiving a data query request, wherein the data query request carries a target data identifier;

[0135] A determination module, configured to determine, according to the target data identifier, a target data query path corresponding to the target data identifier in the correspondence between each data identifier and the data query path included in the data query list;

[0136] The feedback module is used to obtain the target report file data in the second distributed storage space based on the target data query path, and feed it back to the front end in the form of a byte stream.

[0137] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0138] In one embodiment, Fig. 9 As shown, a data processing system 900 is provided, including: a data access unit 901, a data storage unit 902 and a data processing unit 903, wherein:

[0139] The data access unit 901 is used to receive a data stream, which includes a compressed data packet;

[0140] A data storage unit 902, configured to store the data stream in a first distributed storage space based on a preset data storage path;

[0141] The data processing unit 903 is used to scan the data in the stored data stream according to the preset time interval and the preset data storage path, and detect whether there is a signal file in the data stream; if there is a signal file, the compressed data packet in the data stream is decompressed, and the decompressed data is stored in the second distributed storage space based on the preset data query path.

[0142] In one embodiment, the data processing system 900 further includes:

[0143] The report data application unit is used to receive a data query request, and determine a target query path corresponding to the target data identifier in the corresponding relationship between each data identifier information and the data query path contained in the data query list according to the target data identifier carried in the data query request;

[0144] Based on the target query path, the target report file data is obtained in the second distributed storage space and fed back to the front end in the form of a byte stream.

[0145] In one of the embodiments, the data processing unit 903 includes a scheduling module and a resource allocation module;

[0146] A scheduling module, used to scan the data in the stored data stream according to a preset time interval and a preset data storage path, and detect whether there is a signal file in the data stream;

[0147] The resource allocation module is used to allocate data processing threads. If a signal file exists, the compressed data packets in the data stream are decompressed by the data processing threads, and the decompressed data is stored in the second distributed storage space based on a preset data query path.

[0148] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data query lists and data in accessed data streams. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0149] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0150] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0151] receiving a data stream, the data stream including compressed data packets;

[0152] Based on a preset data storage path, storing the data stream in a first distributed storage space;

[0153] Scan the data in the stored data stream according to the preset time interval and the preset data storage path to detect whether there is a signal file in the data stream;

[0154] If the signal file exists, the compressed data packet in the data stream is decompressed, and the decompressed data is stored in the second distributed storage space based on a preset data query path, so that data query is performed in the second distributed storage space.

[0155] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0156] If a signal file exists, in response to the triggering of the signal file, each compressed data packet is split according to a preset data block size to obtain a plurality of data blocks to be decompressed;

[0157] Splitting each data block to be decompressed according to a preset data fragment size to obtain multiple data fragments to be decompressed;

[0158] By calling the data processing thread, the compressed data in each data slice is parsed and calculated to obtain the decompressed slice data;

[0159] The decompressed segmented data is aggregated and processed to obtain a decompressed report file.

[0160] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0161] Matching a corresponding data query path for each decompressed report file data, and storing the decompressed report file data in a target storage space of the distributed storage space based on each data query path;

[0162] Extract characteristic information of each decompressed report file data and generate data identifier of the report file data;

[0163] A data query list including the correspondence between data identifiers and data query paths is established, and the data query list is pushed to the front end for data query.

[0164] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0165] receiving a data query request, the data query request carrying a target data identifier;

[0166] According to the target data identifier, determine the target data query path corresponding to the target data identifier in the correspondence between each data identifier and the data query path included in the data query list;

[0167] Based on the target data query path, the target report file data is obtained in the second distributed storage space and fed back to the front end in the form of a byte stream.

[0168] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0169] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0170] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0171] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: receiving a data stream, wherein the data stream includes compressed data packets; Based on a preset data storage path, storing the data stream in a first distributed storage space; Scanning the data in the stored data stream according to the preset time interval and the preset data storage path to detect whether there is a signal file in the data stream; If the signal file exists, the compressed data packets in the data stream are decompressed, and the decompressed data are stored in the second distributed storage space based on a preset data query path, so that data query is performed in the second distributed storage space.

2. The method according to claim 1, characterized in that If the signal file exists, decompressing the compressed data packet in the data stream comprises: If the signal file exists, in response to the triggering of the signal file, each compressed data packet is split according to a preset data block size to obtain a plurality of data blocks to be decompressed; Splitting each of the data blocks to be decompressed according to a preset data slice size to obtain a plurality of data slices to be decompressed; By calling the data processing thread, the compressed data in each of the data slices is parsed and calculated to obtain the decompressed slice data; The decompressed segmented data are aggregated to obtain a decompressed report file.

3. The method according to claim 2, characterized in that The step of storing the decompressed data in the second distributed storage space based on the preset data query path includes: Matching a corresponding data query path for each decompressed report file data, and storing the decompressed report file data in a target storage space of the distributed storage space based on each data query path; Extracting characteristic information of each of the decompressed report file data and generating a data identifier of the report file data; A data query list including the correspondence between the data identifier and the data query path is established, and the data query list is pushed to the front end for data query.

4. The method according to claim 3, characterized in that: The method further comprises: receiving a data query request, wherein the data query request carries a target data identifier; According to the target data identifier, determine the target data query path corresponding to the target data identifier in the correspondence between each data identifier and the data query path included in the data query list; Based on the target data query path, the target report file data is obtained in the second distributed storage space and fed back to the front end in the form of a byte stream.

5. A data processing device, characterized in that: The device comprises: A receiving module, used for receiving a data stream, wherein the data stream includes a compressed data packet; A storage module, configured to store the data stream in a first distributed storage space based on a preset data storage path; A detection module, used to scan the data in the stored data stream according to a preset time interval and the preset data storage path, and detect whether there is a signal file in the data stream; A processing module is used to decompress the compressed data packets in the data stream if the signal file exists, and store the decompressed data in the second distributed storage space based on a preset data query path, so as to perform data query in the second distributed storage space.

6. A data processing system, characterized in that: The data processing system comprises: A data access unit, configured to receive a data stream, wherein the data stream includes a compressed data packet; A data storage unit, configured to store the data stream in a first distributed storage space based on a preset data storage path; A data processing unit is used to scan the data in the stored data stream according to a preset time interval and the preset data storage path, and detect whether there is a signal file in the data stream; if the signal file exists, decompress the compressed data packet in the data stream, and store the decompressed data in the second distributed storage space based on the preset data query path.

7. The system according to claim 6, characterized in that The data processing system further comprises: The report data application unit is used to receive a data query request, and determine, according to the target data identifier carried in the data query request, a target query path corresponding to the target data identifier in the correspondence between each data identifier information and the data query path contained in the data query list; Based on the target query path, the target report file data is obtained in the second distributed storage space and fed back to the front end in the form of a byte stream.

8. The system according to claim 6, characterized in that The data processing unit includes a scheduling module and a resource allocation module; The scheduling module is used to scan the data in the stored data stream according to the preset time interval and the preset data storage path, and detect whether there is a signal file in the data stream; The resource allocation module is used to allocate data processing threads. If the signal file exists, the compressed data packets in the data stream are decompressed by the data processing threads, and the decompressed data are stored in the second distributed storage space based on a preset data query path.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images