Method and device for monitoring unstructured data
Through the edge acquisition platform, the problem of excessive resource usage and insufficient timeliness when monitoring unstructured data in the data middle platform in the prior art is solved, and more efficient resource utilization and faster monitoring response are achieved.
Patent Information
- Application Number
- CN202411947855.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-03-25
AI Technical Summary
In the prior art, the data middle platform monitors unstructured data through polling, resulting in excessive use of network resources and processing resources, and it is difficult to detect data changes in time.
The target system events are obtained based on the monitoring directory through the edge acquisition platform, and these events are sent to the data middle platform to update the status of the unstructured data. The platform can also send heartbeat messages to monitor working status and process and optimize event delivery if necessary.
It reduces the resources occupied by the data middle platform for monitoring the status, improves the timeliness of the monitoring status, and enables the data middle platform to obtain unstructured data changes in a timely manner.
Smart Images
Figure CN119690708A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data monitoring technology, and in particular to a method and device for monitoring unstructured data. Background Art
[0002] The data center needs to continuously monitor the status of unstructured data in multiple systems, including but not limited to monitoring the update, deletion, and addition of unstructured data.
[0003] In the prior art, the data center generally monitors the status by continuously polling the unstructured data of each system through the File Transfer Protocol (FTP) to detect whether the unstructured data has been updated, deleted, or added. This method will obviously occupy more network resources and processing resources of the data center, and polling a system often takes a long time, so this method is also difficult to detect changes in the unstructured data in the system in a timely manner, and thus difficult to handle in a timely manner. Summary of the invention
[0004] To this end, this application discloses the following technical solutions:
[0005] A first aspect of the present application provides a method for monitoring unstructured data, comprising:
[0006] The edge collection platform obtains target system events according to the monitoring directory, where the target system events include events related to unstructured data in the target system;
[0007] The edge acquisition platform sends the target system event to the data center to update the status of the unstructured data in the data center.
[0008] Optionally, also include:
[0009] The edge acquisition platform sends a heartbeat message to the data center to monitor the working status of the edge acquisition platform.
[0010] Optionally, the edge collection platform sends the target system event to the data center, including:
[0011] The edge collection platform saves the obtained target system events in a collection list;
[0012] The edge collection platform detects a plurality of target system events in the collection list that meet preset processing conditions;
[0013] The edge acquisition platform processes a plurality of target system events that meet the processing conditions to obtain a processed system event;
[0014] The edge acquisition platform sends the processed system events to the data center.
[0015] Optionally, also include:
[0016] The edge acquisition platform obtains the unstructured data of the target system;
[0017] The edge acquisition platform sends the unstructured data to the data center.
[0018] Optionally, the edge collection platform obtains target system events according to the monitoring directory, including:
[0019] The edge acquisition platform monitors events occurring in the target system;
[0020] When an event related to the unstructured data indicated by the monitoring directory is monitored, the event related to the unstructured data indicated by the monitoring directory is obtained as a target system event.
[0021] A second aspect of the present application provides a device for monitoring unstructured data, which is applied to an edge collection platform, and the device includes:
[0022] An obtaining unit, configured to obtain target system events according to the monitoring directory, wherein the target system events include events related to unstructured data in the target system;
[0023] A sending unit is used to send the target system event to the data center to update the state of the unstructured data in the data center.
[0024] Optionally, the sending unit is further used for:
[0025] Send a heartbeat message to the data center to monitor the working status of the edge acquisition platform.
[0026] Optionally, when the sending unit sends the target system event to the data middle station, it is used to:
[0027] The edge collection platform saves the obtained target system events in a collection list;
[0028] The edge collection platform detects a plurality of target system events in the collection list that meet preset processing conditions;
[0029] The edge acquisition platform processes a plurality of target system events that meet the processing conditions to obtain a processed system event;
[0030] The edge acquisition platform sends the processed system events to the data center.
[0031] Optionally, the obtaining unit is further used to obtain the unstructured data of the target system;
[0032] The sending unit is further used to send the unstructured data to the data middle station.
[0033] Optionally, when the obtaining unit obtains the target system event according to the monitoring directory, it is used to:
[0034] The edge acquisition platform monitors events occurring in the target system;
[0035] When an event related to the unstructured data indicated by the monitoring directory is monitored, the event related to the unstructured data indicated by the monitoring directory is obtained as a target system event.
[0036] The beneficial effects of this application are:
[0037] On the one hand, by monitoring the target system events through the edge collection platform to monitor the status of unstructured data, the data middle platform does not need to constantly poll the unstructured data of each system, thereby reducing the resources of the data middle platform occupied by the monitoring status. On the other hand, this monitoring method enables the data middle platform to obtain corresponding events immediately after the unstructured data of the target system changes, significantly improving the timeliness of the monitored status. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0039] Figure 1 is a flow chart of a method for monitoring unstructured data provided by an embodiment of the present application;
[0040] Figure 2 This is a schematic diagram of an application scenario of a method for monitoring unstructured data provided by an embodiment of the present application;
[0041] Figure 3 It is a structural diagram of a device for monitoring unstructured data provided in an embodiment of the present application. DETAILED DESCRIPTION
[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0043] This application embodiment provides a method for monitoring unstructured data. Figure 1 , is a flow chart of the method, and the method may include the following steps.
[0044] S101, the edge collection platform obtains target system events according to the monitoring directory, and the target system events include events related to unstructured data in the target system.
[0045] The method provided in this embodiment can be applied to Figure 2 The scene shown.
[0046] like Figure 2 As shown, the data center can be used with multiple file systems, such as Figure 2 In the communication connection between file systems 1 to 3, an edge collection platform as a monitoring node can be deployed in each file system, for example Figure 2 Middle edge collection platforms 1 to 3. Each edge collection platform can use the file system where it is located as the target system, execute the method provided in this embodiment, thereby monitoring the target system events of the target system and sending the monitored target system events to the data middle platform.
[0047] The target system may include multiple unstructured data. As an example, the target system may be Figure 2 In any file system, the unstructured data may be file data or document data (such as electronic contract data) stored in the file system.
[0048] Each edge collection platform can be pre-configured with a monitoring directory. The monitoring directory can be configured when the edge collection platform is deployed, or it can be updated on demand by the data center during the operation of the edge collection platform.
[0049] For each edge collection platform, the monitoring directory can be used to indicate which unstructured data events of the target system the edge collection platform needs to monitor. For example, the monitoring directory can record several output paths, and the edge collection platform can send events related to the unstructured data pointed to by these data paths as target system events to the data center, and other events are not sent to the data center as target system events.
[0050] That is, S101 may include:
[0051] The edge acquisition platform monitors events occurring in the target system;
[0052] When an event related to the unstructured data indicated by the monitoring directory is monitored, the event related to the unstructured data indicated by the monitoring directory is obtained as a target system event.
[0053] Events related to unstructured data include, but are not limited to, events of adding new unstructured data, events of deleting unstructured data, events of changing unstructured data (such as changing the file name, editing the file content, etc.), etc.
[0054] As an example, assume that the monitoring directory records file path 1, and the file system contains file 1 stored in file path 1 and file 2 stored in file path 2. When the edge collection platform of the file system monitors a deletion event for file 1, the edge collection platform obtains the event as a target system event and sends the target system event to the data middle station in S102; when the edge collection platform of the file system monitors a change event for file 2, the edge collection platform can ignore the event and does not send the event as a target system event to the data middle station.
[0055] In some optional embodiments, a monitoring directory of an edge collection platform may also include multiple monitoring sub-directories. Similar to the monitoring directory, the monitoring sub-directories may also record the data paths of specific unstructured data that need to be monitored in the target system. The edge collection platform can obtain events related to the unstructured data indicated by these directories as target system events based on the monitoring directory and monitoring sub-directories.
[0056] By setting monitoring directories and monitoring sub-directories, the edge collection platform can monitor the unstructured data under specific paths in the target system without having to monitor all unstructured data stored in the target system. This can reduce the resources occupied by the edge collection platform during operation and improve operating efficiency.
[0057] In some embodiments, a monitoring directory may not be set up. The edge collection platform monitors each event related to unstructured data occurring in the target system. Each time such an event is monitored, the event is obtained as a target system event and sent to the data center.
[0058] S102, the edge collection platform sends the target system event to the data center to update the status of the unstructured data in the data center.
[0059] In step S102, the edge acquisition platform can send the obtained target system events to the data center in real time, that is, every time a target system event is obtained, the target system event is sent to the data center.
[0060] Alternatively, the edge collection platform can also create a collection list. When multiple target system events are obtained continuously, the edge collection platform can first record these target system events in the collection list, and then send the target system events in the list to the data center one by one in order.
[0061] The data center can record information about unstructured data in each system, such as which files each system has, what the file names are, how big the files are, what the file types are, what the current status of the files is, etc. Every time a target system event is received, the status of the corresponding unstructured data can be maintained based on the target system event.
[0062] Combined with the above example, assuming that the data center receives an event that file 1 is deleted from the edge collection platform of file system 1, the status of file 1 recorded in the data center can be updated to the deleted status;
[0063] Assume that the data center receives an event that file 2 has been changed from the edge collection platform of file system 2, and the status of file 2 recorded in the data center can be updated to the changed status;
[0064] Assuming that the data center receives an event of a newly added file 3 from the edge collection platform of file system 3, it can record the newly added file 3 of file system 3 and set the status of file 3 recorded in the data center to newly added.
[0065] The beneficial effects of this embodiment are:
[0066] On the one hand, by monitoring the target system events through the edge collection platform to monitor the status of unstructured data, the data middle platform does not need to constantly poll the unstructured data of each system, thereby reducing the resources of the data middle platform occupied by the monitoring status. On the other hand, this monitoring method enables the data middle platform to obtain corresponding events immediately after the unstructured data of the target system changes, significantly improving the timeliness of the monitored status.
[0067] Moreover, through the solution of this embodiment, the user can traverse each file system that needs to be collected, and deploy edge collection platforms in these file systems. The deployed edge collection platform and the data middle platform can use the producer / consumer model to efficiently collect target system events of the target system without the need for construction, which effectively improves the efficiency of collecting target system events.
[0068] In some optional embodiments, the edge acquisition platform may further perform the following steps:
[0069] The edge collection platform sends heartbeat messages to the data center to monitor the working status of the edge collection platform.
[0070] In this embodiment, multiple message queues can be set up in the data middle platform in advance, each message queue corresponds to an edge collection platform, and a heartbeat service can be deployed on each edge collection platform. Thus, each edge collection platform can send heartbeat messages to the data middle platform at a certain period through the heartbeat service, and the sent heartbeat messages can be recorded in the message queue corresponding to the edge collection platform.
[0071] The data center can detect whether these message queues receive heartbeat messages regularly, thereby determining whether the corresponding edge collection platform can monitor the events of the target system normally.
[0072] If the data center finds that a message queue has not received the regularly sent heartbeat messages, it can determine that an abnormality has occurred in the edge collection platform corresponding to the message queue, and output a corresponding abnormal prompt message to prompt relevant personnel to perform maintenance.
[0073] The beneficial effect of this embodiment is that by deploying a heartbeat service on the edge collection platform to send a heartbeat to a designated message queue, the running status of the edge collection platform can be monitored, thereby enabling the abnormality of the edge collection platform to be discovered in a timely manner.
[0074] In some optional embodiments, as described above, the edge collection platform may save multiple target system events obtained through monitoring in a collection list. In this case, the implementation of step S102 may be:
[0075] The edge collection platform saves the acquired target system events in the collection list;
[0076] The edge collection platform detects multiple target system events in the collection list that meet the preset processing conditions;
[0077] The edge acquisition platform processes multiple target system events that meet the processing conditions to obtain a processed system event;
[0078] The edge collection platform sends the processed system events to the data center.
[0079] Among them, multiple target system events that meet the preset processing conditions may include multiple target system events that are unnecessary in the target system or caused by the system's own mechanism. In this embodiment, the edge acquisition platform can determine multiple target system events in the acquisition list that occur continuously in a short period of time and are related to the same unstructured data (such as the same file or the same document) as multiple target system events that meet the processing conditions. The short time here can be set as needed, for example, defined as within 5 seconds, within 2 seconds, etc., without limitation.
[0080] As an example, when a user creates a new document named "AA Document" in a file system, the file system will continuously generate three target system events in a short period of time: a new document event, a document deletion event, and a document name change event to "AA Document", and these three target system events are all related to the final newly added document named "AA Document". The edge collection platform can monitor the above three target system events and save them in the collection list, and then find that the above three target system events meet the processing conditions.
[0081] The method for processing multiple target system events that meet the processing conditions can be that the edge acquisition platform determines a system event that can reach the same state after execution based on the states of the multiple target system events that meet the processing conditions after execution, and uses the system event as the processed system event.
[0082] After obtaining the processed system event, the edge collection platform can replace multiple target system events in the list that meet the processing conditions with the processed system event, and when sending to the data center, only send this processed system event, and no longer send the processed multiple target system events that meet the processing conditions.
[0083] Combined with the above example, the state after the three target system events in the above example are executed successively is that a new document named "AA document" is added to the target system. Therefore, the edge collection platform can determine a new event of a new document named "AA document" and send the new event as a processed system event to the data middle station to replace the above three target system events that meet the processing conditions.
[0084] Through the above embodiments, the edge acquisition platform can optimize multiple monitored target system events that meet the processing conditions into a processed system event, thereby reducing the number of events that the data center needs to receive and process, and further reducing the resources consumed by the data center.
[0085] In some optional embodiments, the edge acquisition platform may further perform the following steps:
[0086] The edge acquisition platform obtains unstructured data from the target system;
[0087] The edge collection platform sends unstructured data to the data center.
[0088] In this embodiment, the edge acquisition platform can obtain the unstructured data of the target system, and then send the unstructured data of the target system to the data middle platform through the FTP service.
[0089] When the edge collection platform receives the unstructured data sending instruction issued by the data center, it can obtain the unstructured data specified by the instruction and send it to the data center.
[0090] Alternatively, when new unstructured data is added to the target system, the edge collection platform can obtain the newly added unstructured data and send it to the data middle platform.
[0091] Alternatively, after the content of unstructured data in the target system is changed (for example, a document is edited), the edge acquisition platform can obtain the changed unstructured data and send it to the data center.
[0092] Optionally, the edge collection platform can send unstructured data synchronously when sending target system events. For example, when sending an event of newly added unstructured data, the newly added unstructured data will be sent synchronously; when sending an event of changed unstructured data, the changed unstructured data will be sent synchronously.
[0093] The beneficial effect of this embodiment is that it enables the data middle platform to synchronize the unstructured data stored in the target system in a timely manner.
[0094] This embodiment also provides a device for monitoring unstructured data, which is applied to an edge collection platform. Figure 3 , is a schematic diagram of the structure of the device, and the device may include the following units.
[0095] An obtaining unit 301 is used to obtain target system events according to the monitoring directory, where the target system events include events related to unstructured data in the target system;
[0096] The sending unit 302 is used to send the target system event to the data center to update the status of the unstructured data in the data center.
[0097] Optionally, the sending unit 302 is further configured to:
[0098] Send heartbeat messages to the data center to monitor the working status of the edge collection platform.
[0099] Optionally, when the sending unit 302 sends the target system event to the data middle station, it is used to:
[0100] The edge collection platform saves the acquired target system events in the collection list;
[0101] The edge collection platform detects multiple target system events in the collection list that meet the preset processing conditions;
[0102] The edge acquisition platform processes multiple target system events that meet the processing conditions to obtain a processed system event;
[0103] The edge collection platform sends the processed system events to the data center.
[0104] Optionally, the obtaining unit 301 is further used to obtain unstructured data of the target system;
[0105] The sending unit 302 is also used to send the unstructured data to the data middle station.
[0106] Optionally, when the obtaining unit 301 obtains the target system event according to the monitoring directory, it is used to:
[0107] The edge acquisition platform monitors events occurring in the target system;
[0108] When an event related to the unstructured data indicated by the monitoring directory is monitored, the event related to the unstructured data indicated by the monitoring directory is obtained as a target system event.
[0109] The working principle of the device for monitoring unstructured data in this embodiment can be found in the relevant steps of the method for monitoring unstructured data provided in any embodiment of the present application, and will not be repeated here.
[0110] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0111] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0112] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.
[0113] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0114] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for monitoring unstructured data, characterized in that: include: The edge collection platform obtains target system events according to the monitoring directory, where the target system events include events related to unstructured data in the target system; The edge acquisition platform sends the target system event to the data center to update the status of the unstructured data in the data center.
2. The method according to claim 1, characterized in that: Also includes: The edge acquisition platform sends a heartbeat message to the data center to monitor the working status of the edge acquisition platform.
3. The method according to claim 1, characterized in that The edge acquisition platform sends the target system event to the data center, including: The edge collection platform saves the obtained target system events in a collection list; The edge collection platform detects a plurality of target system events in the collection list that meet preset processing conditions; The edge acquisition platform processes a plurality of target system events that meet the processing conditions to obtain a processed system event; The edge acquisition platform sends the processed system events to the data center.
4. The method according to claim 1, characterized in that: Also includes: The edge acquisition platform obtains the unstructured data of the target system; The edge acquisition platform sends the unstructured data to the data center.
5. The method according to claim 1, characterized in that: The edge acquisition platform obtains target system events according to the monitoring directory, including: The edge acquisition platform monitors events occurring in the target system; When an event related to the unstructured data indicated by the monitoring directory is monitored, the event related to the unstructured data indicated by the monitoring directory is obtained as a target system event.
6. A device for monitoring unstructured data, characterized in that: Applied to the edge acquisition platform, the device includes: An obtaining unit, configured to obtain target system events according to the monitoring directory, wherein the target system events include events related to unstructured data in the target system; A sending unit is used to send the target system event to the data center to update the state of the unstructured data in the data center.
7. The device according to claim 6, characterized in that The sending unit is also used for: Send a heartbeat message to the data center to monitor the working status of the edge acquisition platform.
8. The device according to claim 6, characterized in that When the sending unit sends the target system event to the data center, it is used to: The edge collection platform saves the obtained target system events in a collection list; The edge collection platform detects a plurality of target system events in the collection list that meet preset processing conditions; The edge acquisition platform processes a plurality of target system events that meet the processing conditions to obtain a processed system event; The edge acquisition platform sends the processed system events to the data center.
9. The device according to claim 6, characterized in that The obtaining unit is further used to obtain the unstructured data of the target system; The sending unit is further configured to send the unstructured data to the data middle station.
10. The device according to claim 6, characterized in that When the obtaining unit obtains the target system event according to the monitoring directory, it is used to: The edge acquisition platform monitors events occurring in the target system; When an event related to the unstructured data indicated by the monitoring directory is monitored, the event related to the unstructured data indicated by the monitoring directory is obtained as a target system event.