Data analysis acceleration and optimization processing method based on data service area

Abstract

The invention relates to a data analysis acceleration and optimization processing method based on a data service area. The method comprises the following steps: step S100, acquiring monitoring data from an acquisition terminal arranged in a data service area; s200, splitting the monitoring data into first-level data, second-level data and third-level data according to a preset data grading relationship; s300, the first-level data, the second-level data and the third-level data are stored in a preset first-level storage space, a preset second storage space and a preset third-level storage space respectively; and S400, performing data integrity verification in the third-level storage space, and deleting the monitoring data after the verification is completed. By means of the method, the accuracy of data storage in the data distributed storage process can be improved.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a data analysis acceleration and optimization processing method, system, computer equipment, storage medium and computer program product based on a data service area. Background Art

[0002] With the continuous advancement of industrial informatization, how to manage massive industrial data has become the primary issue in further promoting the informatization of industrial scenarios.

[0003] The existing Chinese invention patent with publication number CN118467212A discloses a method and system for processing industrial equipment data, including the following steps: step S1, formulate a quick analysis and processing method during the industrial equipment data analysis and processing process; step S2, monitor the industrial equipment data processing results, and judge the analysis abnormality; step S3, when the analysis abnormality occurs, adjust the data analysis mode, and perform self-repair of the data analysis abnormality; step S4, output the abnormal nodes in the quick analysis processing, and manually optimize the quick analysis processing method; step S5, according to the human resource allocation, adjust the analysis resource allocation of each industrial equipment, and limit the fastest analysis processing speed to be within the operational range of human resources. The present invention has the characteristics of intelligent switching processing and strong practicality.

[0004] However, the above-mentioned industrial equipment data processing method and system can only check and process the current anomalies at the industrial site, but still cannot guarantee that the repaired data can be accurately saved, especially in the scenario where industrial data needs to be stored in a distributed manner. It is even more difficult to avoid data errors during data transmission. Summary of the invention

[0005] In a first aspect, the present application provides a data analysis acceleration and optimization processing method based on a data service area, the method comprising: Acquire monitoring data from a collection terminal located in the data service area; According to the preset data classification relationship, the monitoring data is divided into first-level data, second-level data and third-level data; The first-level data, the second-level data and the third-level data are stored in the preset first-level storage space, the second-level storage space and the third-level storage space respectively; Data integrity verification is performed in the third-level storage space and the monitoring data is deleted after completion.

[0006] In one of the embodiments, the monitoring data includes parameter change data and time stamp data corresponding to the parameter change data one by one; According to the preset data classification relationship, the specific steps of splitting the monitoring data into first-level data, second-level data and third-level data include: Extracting a plurality of timestamp data with equal time intervals from the monitoring data as sampling timestamp data according to a preset first sampling frequency; Extracting a number of parameter change data corresponding to the sampling timestamp data one by one as sampling parameter data; The sampling parameter data and the sampling timestamp data are matched one by one and stored to form the first level data.

[0007] In one embodiment, the specific steps of splitting the monitoring data into first-level data, second-level data, and third-level data according to the preset data classification relationship include: Extract inflection point data and its corresponding timestamp data from parameter change data; Removing the sampling timestamp data from the timestamp data corresponding to the inflection point data, and using the remaining timestamp data and the inflection point data corresponding thereto as the second-level data; The first-level data and the second-level data are removed from the monitoring data, and the remaining monitoring data is used as the third-level data.

[0008] In one embodiment, the third-level storage space includes a data verification temporary storage area; The specific steps for verifying data integrity in the third-level storage space and deleting the monitoring data after completion include: Extract the first-level data and the second-level data and store them in the data verification temporary storage area; The third-level data is stored in the data verification temporary storage area, and the first-level data and the second-level data are merged according to the timestamp data to obtain the data to be verified; The data to be verified is matched with the monitoring data through random sampling. If both are matched successfully, it means that the data verification is passed, and the monitoring data and the data to be verified in the data verification temporary storage area are deleted. Only the first-level data, second-level data and third-level data are saved in the first-level storage space, second-level storage space and third-level storage space respectively.

[0009] In one embodiment, the method further comprises: Waiting and determining whether data access information is obtained, the access information including access permission registration information and data access requirement information; Reading data access requirement information and determining whether the accessed data belongs to first-level data, second-level data or third-level data; Determine whether the access permission level is higher than the data registration to which the accessed data belongs; If so, the first level data, the second level data or the third level data corresponding to the data access requirement information is output.

[0010] In a second aspect, the present application also provides a data analysis acceleration and optimization processing system based on a data service area, the device comprising: Collection terminal, used to collect monitoring data and output; Data reading and writing module, used to obtain and temporarily store monitoring data, and classify the monitoring data; A storage module, used to provide a first-level storage space, a second-level storage space and a third-level storage space to store first-level data, second-level data and third-level data; The data verification module is used to verify the integrity of the monitoring data.

[0011] In one embodiment, the data reading and writing module includes: The data writing submodule is used to write the monitoring data obtained from the collection terminal into the storage module; The data reading submodule is used to read the required monitoring data from the storage module according to the acquired data access information.

[0012] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program: Acquire monitoring data from a collection terminal located in the data service area; According to the preset data classification relationship, the monitoring data is divided into first-level data, second-level data and third-level data; The first-level data, the second-level data and the third-level data are stored in the preset first-level storage space, the second-level storage space and the third-level storage space respectively; Data integrity verification is performed in the third-level storage space and the monitoring data is deleted after completion.

[0013] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Acquire monitoring data from a collection terminal located in the data service area; According to the preset data classification relationship, the monitoring data is divided into first-level data, second-level data and third-level data; The first-level data, the second-level data and the third-level data are stored in the preset first-level storage space, the second-level storage space and the third-level storage space respectively; Data integrity verification is performed in the third-level storage space and the monitoring data is deleted after completion.

[0014] In a fifth aspect, the present application further provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following steps are implemented: Acquire monitoring data from a collection terminal located in the data service area; According to the preset data classification relationship, the monitoring data is divided into first-level data, second-level data and third-level data; The first-level data, the second-level data and the third-level data are stored in the preset first-level storage space, the second-level storage space and the third-level storage space respectively; Data integrity verification is performed in the third-level storage space and the monitoring data is deleted after completion.

[0015] The above-mentioned data analysis acceleration and optimization processing method, device, computer equipment, storage medium and computer program product based on the data service area, by reintegrating and verifying the split first-level data, second-level data and third-level data, to ensure that the split first-level data, second-level data and third-level data can match the original detection data, thereby improving the accuracy of the data distributed storage process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 An application environment diagram of a data analysis acceleration and optimization processing method based on a data service area in one embodiment; Figure 2 A schematic diagram of a flow chart of a data analysis acceleration and optimization processing method based on a data service area in one embodiment; Figure 3 is a flow chart of step S200 of an embodiment; Figure 4 It is a structural block diagram of a data analysis acceleration and optimization processing system based on a data service area in one embodiment; Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] The data analysis acceleration and optimization processing method based on the data service area provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0019] In one embodiment, Figure 2 As shown, a data analysis acceleration and optimization processing method based on a data service area is provided, and the method is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps: In a first aspect, the present application provides a data analysis acceleration and optimization processing method based on a data service area, the method comprising: Step S100: Acquire monitoring data from a collection terminal located in a data service area.

[0020] The monitoring data includes parameter change data and timestamp data corresponding to the parameter change data.

[0021] Step S200: split the monitoring data into first-level data, second-level data and third-level data according to the preset data classification relationship.

[0022] The first level data, the second level data and the third level data are respectively data levels preset by the system and used to distinguish data with different access rights. The specific steps of step S200 include: Step S210: extracting a plurality of timestamp data with equal time intervals from the monitoring data as sampling timestamp data according to a preset first sampling frequency.

[0023] Step S220: extracting a plurality of parameter change data corresponding one-to-one to the sampling timestamp data as sampling parameter data.

[0024] Step S230: The sampling parameter data and the sampling timestamp data are matched one by one and stored to form the first level data.

[0025] Through steps S210-S230, the monitoring data can be segmented according to time. The timestamp data corresponding to the segmentation point and its corresponding reference data are the required first-level data. The first-level data is also the data with the lowest required access permission, that is, the visitor can only access part of the data extracted from the detection data at equal time intervals with the lowest level of access permission.

[0026] Step S240: extracting the inflection point data and its corresponding timestamp data from the parameter change data.

[0027] Step S250: removing the sampling timestamp data from the timestamp data corresponding to the inflection point data, and taking the remaining timestamp data and the inflection point data corresponding thereto as the second-level data.

[0028] Step S260: remove the first-level data and the second-level data from the monitoring data, and use the remaining monitoring data as the third-level data.

[0029] Through steps S240-S260, the inflection point data in the monitoring data can be extracted for the second time, and the overlapping data between the inflection point data and the first-level data needs to be excluded. This part of the data is used as the second-level data, so that the inflection point data that can reflect the important data change nodes in the detection data can be accessed through the second-level access rights. Compared with the first-level data, the second-level data can reflect more details of the detection data changes, and at the same time, the access rights required for this part of the detection data are improved, and the risk of industrial data loss can also be reduced. Other data in the detection data except the first-level data and the second-level data can present all parameter changes of the detected object during the monitoring process. Giving the highest access rights requirements to this part of the data can also further reduce the risk of data loss.

[0030] Step S300: storing the first-level data, the second-level data and the third-level data in a preset first-level storage space, a second-level storage space and a third-level storage space respectively.

[0031] Step S400: Perform data integrity verification in the third-level storage space and delete the monitoring data after completion.

[0032] In this embodiment, the third-level storage space includes a data verification temporary storage area. The specific steps of step S400 include: Step S410: extracting the first-level data and the second-level data, and storing them in the data verification temporary storage area; Step S420: storing the third-level data in the data verification temporary storage area, and merging the first-level data and the second-level data according to the timestamp data to obtain the data to be verified; Step S430: Match the data to be verified with the monitoring data by random sampling. If both are matched successfully, it means that the data verification is passed. The monitoring data and the data to be verified in the data verification temporary storage area are deleted, and only the first-level data, second-level data and third-level data are saved in the first-level storage space, the second-level storage space and the third-level storage space respectively.

[0033] Through steps S410-S430, the first-level data, second-level data and third-level data can be restored to the data to be verified in the data verification temporary storage area. If there is no error or omission in the data during the transmission process, the data to be verified should be the same as the monitoring data. Therefore, the monitoring data is matched with the data to be verified. If the match is successful, it means that there is no error or omission in the data.

[0034] Step S500: Wait and determine whether data access information is obtained, where the access information includes access permission registration information and data access requirement information.

[0035] Step S600: Read data access requirement information and determine whether the accessed data belongs to first-level data, second-level data or third-level data.

[0036] Step S700: Determine whether the access permission level is higher than the data registration to which the accessed data belongs.

[0037] Step S800: If yes, output the first level data, the second level data or the third level data corresponding to the data access requirement information.

[0038] Step S900: If not, output a prompt message indicating insufficient access rights.

[0039] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0040] Based on the same inventive concept, the embodiment of the present application also provides a data analysis acceleration and optimization processing system based on a data service area for implementing the data analysis acceleration and optimization processing method based on a data service area involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the data analysis acceleration and optimization processing system based on a data service area provided below can refer to the limitations of the data analysis acceleration and optimization processing method based on a data service area above, and will not be repeated here.

[0041] In one embodiment, Figure 5 As shown, a data analysis acceleration and optimization processing system based on a data service area is provided, including: a collection terminal, a data reading and writing module, a storage module and a data verification module, wherein: Collection terminal, used to collect monitoring data and output; Data reading and writing module, used to obtain and temporarily store monitoring data, and classify the monitoring data; A storage module, used to provide a first-level storage space, a second-level storage space and a third-level storage space to store first-level data, second-level data and third-level data; The data verification module is used to verify the integrity of the monitoring data.

[0042] Among them, the data reading and writing module also includes: The data writing submodule is used to write the monitoring data obtained from the collection terminal into the storage module; The data reading submodule is used to read the required monitoring data from the storage module according to the acquired data access information.

[0043] Each module in the data analysis acceleration and optimization processing system based on the data service area can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0044] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data analysis acceleration and optimization processing method based on a data service area is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0045] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0046] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented: Step S100: Acquire monitoring data from a collection terminal located in a data service area.

[0047] Step S200: split the monitoring data into first-level data, second-level data and third-level data according to the preset data classification relationship.

[0048] Step S300: storing the first-level data, the second-level data and the third-level data in a preset first-level storage space, a second-level storage space and a third-level storage space respectively.

[0049] Step S400: Perform data integrity verification in the third-level storage space and delete the monitoring data after completion.

[0050] Step S500: Wait and determine whether data access information is obtained, where the access information includes access permission registration information and data access requirement information.

[0051] Step S600: Read data access requirement information and determine whether the accessed data belongs to first-level data, second-level data or third-level data.

[0052] Step S700: Determine whether the access permission level is higher than the data registration to which the accessed data belongs.

[0053] Step S800: If yes, output the first level data, the second level data or the third level data corresponding to the data access requirement information.

[0054] Step S900: If not, output a prompt message indicating insufficient access rights.

[0055] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Step S100: Acquire monitoring data from a collection terminal located in a data service area.

[0056] Step S200: split the monitoring data into first-level data, second-level data and third-level data according to the preset data classification relationship.

[0057] Step S300: storing the first-level data, the second-level data and the third-level data in a preset first-level storage space, a second-level storage space and a third-level storage space respectively.

[0058] Step S400: Perform data integrity verification in the third-level storage space and delete the monitoring data after completion.

[0059] Step S500: Wait and determine whether data access information is obtained, where the access information includes access permission registration information and data access requirement information.

[0060] Step S600: Read data access requirement information and determine whether the accessed data belongs to first-level data, second-level data or third-level data.

[0061] Step S700: Determine whether the access permission level is higher than the data registration to which the accessed data belongs.

[0062] Step S800: If yes, output the first level data, the second level data or the third level data corresponding to the data access requirement information.

[0063] Step S900: If not, output a prompt message indicating insufficient access rights.

[0064] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps: Step S100: Acquire monitoring data from a collection terminal located in a data service area.

[0065] Step S200: split the monitoring data into first-level data, second-level data and third-level data according to the preset data classification relationship.

[0066] Step S300: storing the first-level data, the second-level data and the third-level data in a preset first-level storage space, a second-level storage space and a third-level storage space respectively.

[0067] Step S400: Perform data integrity verification in the third-level storage space and delete the monitoring data after completion.

[0068] Step S500: Wait and determine whether data access information is obtained, where the access information includes access permission registration information and data access requirement information.

[0069] Step S600: Read data access requirement information and determine whether the accessed data belongs to first-level data, second-level data or third-level data.

[0070] Step S700: Determine whether the access permission level is higher than the data registration to which the accessed data belongs.

[0071] Step S800: If yes, output the first level data, the second level data or the third level data corresponding to the data access requirement information.

[0072] Step S900: If not, output a prompt message indicating insufficient access rights.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0074] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0075] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data analysis acceleration and optimization processing method based on a data service area, characterized in that: The method comprises: Acquire monitoring data from a collection terminal located in the data service area; According to the preset data classification relationship, the monitoring data is divided into first-level data, second-level data and third-level data; The first-level data, the second-level data and the third-level data are stored in the preset first-level storage space, the second-level storage space and the third-level storage space respectively; Data integrity verification is performed in the third-level storage space and the monitoring data is deleted after completion.

2. The method according to claim 1, characterized in that: The monitoring data includes parameter change data and timestamp data corresponding to the parameter change data one by one; The specific steps of splitting the monitoring data into first-level data, second-level data and third-level data according to the preset data classification relationship include: Extracting a plurality of timestamp data with equal time intervals from the monitoring data as sampling timestamp data according to a preset first sampling frequency; Extracting a number of parameter change data corresponding to the sampling timestamp data one by one as sampling parameter data; The sampling parameter data and the sampling timestamp data are matched one by one and stored to form the first level data.

3. The method according to claim 2, characterized in that The specific steps of splitting the monitoring data into first-level data, second-level data and third-level data according to the preset data classification relationship include: Extract inflection point data and its corresponding timestamp data from parameter change data; Removing the sampling timestamp data from the timestamp data corresponding to the inflection point data, and using the remaining timestamp data and the inflection point data corresponding thereto as the second-level data; The first-level data and the second-level data are removed from the monitoring data, and the remaining monitoring data is used as the third-level data.

4. The method according to claim 1, characterized in that: The third-level storage space includes a data verification temporary storage area; The specific steps of performing data integrity verification in the third-level storage space and deleting the monitoring data after completion include: Extract the first-level data and the second-level data and store them in the data verification temporary storage area; The third-level data is stored in the data verification temporary storage area, and the first-level data and the second-level data are merged according to the timestamp data to obtain the data to be verified; The data to be verified is matched with the monitoring data through random sampling. If both are matched successfully, it means that the data verification is passed, and the monitoring data and the data to be verified in the data verification temporary storage area are deleted. Only the first-level data, second-level data and third-level data are saved in the first-level storage space, second-level storage space and third-level storage space respectively.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Waiting and determining whether data access information is obtained, the access information including access permission registration information and data access requirement information; Reading data access requirement information and determining whether the accessed data belongs to first-level data, second-level data or third-level data; Determining whether the access permission level is higher than the data registration to which the accessed data belongs; If so, the first level data, the second level data or the third level data corresponding to the data access requirement information is output.

6. A data analysis acceleration and optimization processing system based on a data service area, characterized in that: The system comprises: Collection terminal, used to collect monitoring data and output; Data reading and writing module, used to obtain and temporarily store monitoring data, and classify the monitoring data; A storage module, used to provide a first-level storage space, a second-level storage space and a third-level storage space to store first-level data, second-level data and third-level data; The data verification module is used to verify the integrity of the monitoring data.

7. The system according to claim 6, characterized in that The data reading and writing module comprises: The data writing submodule is used to write the monitoring data obtained from the collection terminal into the storage module; The data reading submodule is used to read the required monitoring data from the storage module according to the acquired data access information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images