Data management method and device, equipment and readable medium

Abstract

The invention provides a data management method, device and equipment and a readable medium, and belongs to the technical field of data processing. The method comprises the steps of obtaining a target data file; determining at least one label corresponding to the target data file, and storing a mapping relationship between the target data file and the at least one label; and managing the target data file based on the stored mapping relationship between the target data file and the at least one label. The method is used for solving the problems that a big data management system is single in data management mode, management and use are lack of flexibility, and diversified requirements of users cannot be met.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data management method, device, equipment and readable medium. Background Art

[0002] At present, in the big data storage system facing multiple data sources, various heterogeneous data are stored and managed in a unified manner. The big data storage system mainly stores and manages data in a unified manner through data cleaning, and displays data through business classification, etc. If a user needs to use data in the big data storage system, he can find the data by searching for the business type. It can be seen that the current big data management system has a single way of managing data, lacks flexibility in management and use, and cannot meet the diverse needs of users. Summary of the invention

[0003] The present disclosure provides a data management method, apparatus, device and readable medium.

[0004] A first aspect of the present disclosure provides a data management method, comprising:

[0005] Get the target data file;

[0006] Determine at least one tag corresponding to the target data file, and store a mapping relationship between the target data file and the at least one tag;

[0007] The target data file is managed based on the stored mapping relationship between the target data file and the at least one tag.

[0008] A second aspect of the present disclosure provides a data management device, including:

[0009] An acquisition module is used to acquire a target data file;

[0010] A processing module, configured to determine at least one tag corresponding to the target data file, and store a mapping relationship between the target data file and the at least one tag;

[0011] A management module is used to manage the target data file based on the stored mapping relationship between the target data file and the at least one tag.

[0012] A third aspect of the present disclosure provides an electronic device, including:

[0013] at least one processor;

[0014] A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to the first aspect;

[0015] At least one I / O interface is connected between the processor and the memory and is configured to implement information interaction between the processor and the memory.

[0016] A fourth aspect of the present disclosure provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to the first aspect is implemented.

[0017] The present disclosure has the following advantages:

[0018] A target data file is obtained and at least one tag corresponding to the target data file is determined, a mapping relationship between the target data file and the at least one tag is obtained and the mapping relationship is stored, so that the target data file can be managed based on the stored mapping relationship later. The tag-based data management method can be compatible with data files with different structures. By flexibly adjusting the tags corresponding to the target data files, the purpose of meeting the diverse needs of users can be achieved. The management method is flexible and efficient, and refined management of target data files can be achieved through tags. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a data management application scenario provided in an embodiment of the present disclosure;

[0020] Figure 2 A flowchart of a data management method provided in an embodiment of the present disclosure is shown;

[0021] Figure 3 A schematic diagram of a process for managing data files in a data management system provided in an embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram of functional division of a data management system provided in an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of the structure of a data management device provided in an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0026] As used in this disclosure, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0027] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure.As used in the present disclosure, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0028] When the terms “comprising” and / or “made of…” are used in the present disclosure, it specifies the existence of the stated features, integers, steps, operations, elements and / or components, but does not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0029] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as those commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this disclosure.

[0030] The embodiments of the present disclosure provide a data management method, which can be applied to any electronic device and is used in scenarios where a large number of data files need to be managed. In this scenario, a large number of data files are uniformly stored and managed, and users analyze and use the stored data files according to business needs. For example, the method is applied to the management of various types of data files such as communication network data, user operation data, and log data. Figure 1 The figure is a schematic diagram of a data management application scenario in an embodiment of the present disclosure. In the application scenario, a large number of data files are stored in the data management system, and users query and use the required data files from the data management system.

[0031] Figure 2 The figure is a flow chart of the data management method provided by the embodiment of the present disclosure, which mainly includes the following steps:

[0032] Step 201, obtaining a target data file.

[0033] In some embodiments, the target data file may be an unstructured data file such as an audio file, a video file, a picture file, a log file, a binary file, or a structured data file. It should be understood that for ease of description, only the target data file is used as an example for description, and any data file can be managed according to the data management method provided in the embodiment of the present disclosure.

[0034] In some embodiments, the target data file may be imported through a data interface, may be imported by user operation through a human-computer interaction interface, may be imported by a third-party system through a corresponding data interface, or may be regularly collected from a third-party system.

[0035] Step 202: determine at least one tag corresponding to the target data file, and store a mapping relationship between the target data file and the at least one tag.

[0036] In some embodiments, the determining of at least one tag corresponding to the target data file includes but is not limited to the following implementations:

[0037] Implementation method 1: obtaining label data imported corresponding to the target data file; parsing the label data to obtain at least one label as at least one label corresponding to the target data file.

[0038] The method of obtaining the label data imported corresponding to the target data file includes but is not limited to the following implementation methods:

[0039] Mode a, obtaining configuration data imported corresponding to the target data file, wherein the configuration data includes tag data configured corresponding to the target data file; and extracting at least one tag corresponding to the target data file from the configuration data.

[0040] In some embodiments, the acquiring the imported configuration data corresponding to the target data file includes: acquiring interactive interface input information corresponding to the target data file, and extracting the label data from the interactive interface input information.

[0041] Mode b, obtaining label data corresponding to the target data file from a third-party system interface, for example, importing a data file from a third-party system interface periodically or once, and obtaining the imported label or the label defined by the file while importing.

[0042] Implementation method 2 is to parse the target data file, extract at least one tag indication data from the target data file, and obtain at least one tag corresponding to the tag indication data as at least one tag corresponding to the target data file.

[0043] In some embodiments, the at least one tag is of at least one of the following tag types:

[0044] Basic feature type labels;

[0045] Business type label;

[0046] Custom type tags.

[0047] Among them, the basic feature type label indicates the basic attributes of the target data file; the business type label indicates the various business types applicable to the target data file, which is set according to business needs; the custom type label indicates various user-defined labels, which can be any label that can meet user needs and is set according to user usage requirements.

[0048] In an exemplary embodiment, when a target data file is imported, a basic feature type label of the target data file is obtained, including at least one of the generation time of the target data file, the generation location of the target data file, the source of the target data, etc. The basic feature type label may also be called a fact label.

[0049] In an exemplary embodiment, while or after importing a target data file, a user-defined type tag configured for the target data file is obtained. The user-defined type tag may be any data that meets the user's own needs, so as to facilitate subsequent classification and retrieval of data files.

[0050] In an exemplary embodiment, during the import process of the target data file, the target data file is parsed in real time, and the imported target data file is labeled with a corresponding business type label according to the business analysis function in the data management system. For example, the cell type and number of cells in the target data file are analyzed and used as the label of the target data file.

[0051] Step 203: managing the target data file based on the stored mapping relationship between the target data file and the at least one tag.

[0052] In some embodiments, the target data file is managed based on the stored mapping relationship between the target data file and the at least one tag, including: obtaining the tag to be queried, obtaining the target data file corresponding to the tag to be queried based on the stored mapping relationship between the target data file and the at least one tag, and performing relevant operations on the target data file.

[0053] In the exemplary embodiment, the user can query the data files stored in the data management system through tags, and modify, download, analyze and other operations on the queried data, and can also modify the corresponding tags of the data files to better meet their business needs. In addition, the user can also browse the data files uploaded by other users through the data management system, and if there is data that meets their own needs, they can also add relevant tags to achieve the purpose of data reuse.

[0054] In some embodiments, for newly imported data files, deduplication processing is performed before storage to avoid repeated storage of the same data files, specifically including: comparing the newly imported data file with each stored data file; when the newly imported data file is identical to any of the stored data files, comparing at least one label corresponding to the newly imported data file with an existing label of the same stored data file, obtaining a target label that is different from the existing label in the at least one label corresponding to the newly imported data file, storing the target label in at least one label corresponding to the same stored data file, and discarding the newly imported data file; when there is no data file identical to the newly imported data file in the stored data files, storing the newly imported data file and the corresponding at least one label.

[0055] The at least one tag corresponding to the newly imported data file refers to one tag or multiple tags corresponding to the newly imported data file, that is, the newly imported data file may correspond to one tag or may correspond to multiple tags.

[0056] Among them, the newly imported data file is compared with each stored data file respectively. There can be many comparison methods, and the specific comparison method adopted is not limited here. For example, the hash value of the newly imported data file is calculated, and the hash value corresponding to each stored data file is calculated respectively. By comparing the hash value of the newly imported data file and the hash value corresponding to each stored data file, if there is a stored data file with the same hash value as the newly imported data file, the newly imported data file is not uploaded, and a prompt message of successful upload is directly returned to the user device, and the tags not included in the existing tags of the stored data file are newly stored for the user to use in subsequent queries.

[0057] In some embodiments, newly added tag identification rules can be used to perform batch analysis on stored data files to identify new tags, specifically including: when a trigger condition is met, tag identification is performed on at least one stored data file according to the newly added tag identification rule to obtain a new tag corresponding to each of the at least one data file; the new tag is added to the existing tag of the corresponding data file in the mapping relationship; wherein the trigger condition includes at least one of the following: arrival of a timing moment; detection of a trigger event.

[0058] For the stored data files, the managed data files are analyzed in batches by setting a timer or triggering an event, and new business tags are added to the data files according to the analysis results. For example, the triggering event may be the detection of a new script program running, which may be developed and uploaded by the user, so that tags that meet specific needs are obtained through the script program, and the data files are analyzed and processed in batches by executing the script program.

[0059] The reaching of the timing time is the time set in the timer, and the current time reaching the time set in the timer is used as a trigger condition.

[0060] In an exemplary embodiment, taking the use of network element data by users in a communication network scenario as an example, the user may need to analyze the network element data to confirm various states of the data, such as the number of cells, the number of neighboring cells, and the special configuration of the network element, in order to determine the use scenarios of the data, which change rapidly according to the needs. In a traditional big data management system, developers create corresponding data warehouses, data marts, etc. to further analyze the data files, and manage them through relational databases. If expansion is required, corresponding development is required, which is difficult to meet the rapidly changing needs of users. The method provided in the embodiment of the present disclosure adds data analysis on the basis of the traditional big data management method, and allows users to upload script programs that meet the requirements. The script program can retrieve and analyze the data files stored in the data management system, and process the analysis results according to user needs, such as pushing the data files that meet the analysis requirements to a server outside the data management system specified by the user, or adding the new tags obtained by the analysis to the corresponding data files. In the method provided in the embodiment of the present disclosure, some business analysis scripts can also be built into the data management system, and the data management system can be set to execute the script program in real time or regularly.

[0061] In an exemplary embodiment, taking the use of network element data by a user in a communication network scenario as an example, the user uploads a copy of network element configuration data to the data management system for use in subsequent testing. In a traditional big data management system, the system administrator first develops a corresponding classification function for network element configuration data to classify the network element configuration data, such as classifying it according to operator information, time information, network element configuration data information, etc., and then filters it according to the classification when the user uses it later. By adopting the method provided in the embodiment of the present disclosure, when uploading the network element configuration data, the user obtains the label that the network element configuration data is marked with, such as operator information, time information, network element configuration data information, etc. In addition, more flexible test scenario labels can also be marked, such as: functional testing, integration testing, performance testing, etc., without waiting for the system administrator to develop the corresponding classification function, and the flexible management of data can be quickly realized. In subsequent retrieval, the corresponding data can be obtained by combining a single label or multiple labels to meet the user's usage scenario requirements.

[0062] In an exemplary embodiment, Figure 3 The figure shows a flow chart of managing data files in a data management system, which mainly includes:

[0063] Step 301, importing a data file into the data management system, and labeling the data file with a fact tag during import;

[0064] Step 302: during the import process, the data file is analyzed in real time and a business tag is added;

[0065] Step 303, triggering a batch analysis process for the data files in the data management system according to a planned timing or on demand, obtaining a new label, and marking the batch processed data files with the new label;

[0066] Step 304: Obtain the tag that the user needs to query and obtain the data file corresponding to the tag.

[0067] In an exemplary embodiment, the data management system is functionally divided according to the data management method described above. Figure 4 The following is a diagram showing the functional division of the data management system, which mainly includes:

[0068] Tag management function: This function is mainly used to manage the tags managed in the data management system, including tag operations such as adding, deleting, modifying, querying, etc., as well as storage management of data according to different tag types.

[0069] Data analysis function: This function is mainly used to analyze data files managed in the data management system. The analysis requirements include streaming analysis and batch analysis. The data analysis function is also more flexible, allowing users to upload scripts for analyzing data files to meet the user's diverse data analysis needs. The data analysis function will analyze the data files and add richer labels to the data according to the needs of the analysis function to meet the user's diverse data demand scenarios.

[0070] Data storage management function: This function mainly meets the storage management needs of data files, including multiple copy storage of data files to ensure data security, and basic operation functions such as importing, applying, viewing, and downloading data files.

[0071] In the disclosed embodiment, a target data file is obtained and at least one tag corresponding to the target data file is determined, a mapping relationship between the target data file and the at least one tag is obtained and stored, so that the target data file is subsequently managed based on the stored mapping relationship. The tag-based data management method is compatible with data files with different structures. By flexibly adjusting the tags corresponding to the target data files, the purpose of meeting the diverse needs of users can be achieved. The management method is flexible and efficient, and the target data files can be managed in a refined manner through tags. By labeling the same data file with different tags, user queries are performed based on the tags, which can meet the needs of users in different scenarios and realize the reuse of data files.

[0072] The method provided by the embodiment of the present disclosure is applied to the communication field, which can meet the needs of managing network data, system data, log data, etc. in the research and development and production environments in the communication field. In addition, the method provided by the embodiment of the present disclosure is applicable to scenarios where a large amount of data needs to be managed.

[0073] Querying data files by tags can supplement existing search query methods and provide richer search methods, allowing users to quickly locate data files that meet their needs.

[0074] The step division of the above various methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this disclosure. Adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of this disclosure.

[0075] A data management device is provided in an embodiment of the present disclosure. The specific implementation of the device can refer to the relevant description of the method embodiment, which will not be repeated here. Figure 5The schematic diagram of the structure of the device is shown, which mainly includes:

[0076] An acquisition module 501 is used to acquire a target data file;

[0077] A processing module 502 is used to determine at least one tag corresponding to the target data file, and store a mapping relationship between the target data file and the at least one tag;

[0078] The management module 503 is used to manage the target data file based on the stored mapping relationship between the target data file and the at least one tag.

[0079] The functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the method embodiments. The specific implementation and technical effects thereof can be referred to the description of the method embodiments above, and will not be described again here for the sake of brevity.

[0080] It should be noted that all modules involved in this embodiment are logic modules. In practical applications, a logic unit may be a physical unit, or a part of a physical unit, or may be implemented as a combination of multiple physical units. In addition, in order to highlight the innovative part of the present disclosure, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by the present disclosure, but this does not mean that there are no other units in this embodiment.

[0081] Reference Figure 6 , an embodiment of the present disclosure provides an electronic device, comprising:

[0082] at least one processor 601;

[0083] A memory 602 having at least one program stored thereon, and when the at least one program is executed by the at least one processor, the at least one processor implements the above method;

[0084] At least one I / O interface 603 is connected between the processor and the memory and is configured to implement information exchange between the processor and the memory.

[0085] Among them, the processor 601 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 602 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 603 is connected between the processor 601 and the memory 602, and can realize information interaction between the processor 601 and the memory 602, including but not limited to a data bus (Bus), etc.

[0086] In some embodiments, the processor 601 , the memory 602 , and the I / O interface 603 are connected to each other through a bus, and further connected to other components of the computing device.

[0087] This embodiment further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in this embodiment is implemented. To avoid repeated description, the specific steps of the method are not repeated here.

[0088] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods applied for above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0089] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0090] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present embodiment and to form different embodiments.

[0091] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A data management method, characterized in that: include: Get the target data file; Determine at least one tag corresponding to the target data file, and store a mapping relationship between the target data file and the at least one tag; The target data file is managed based on the stored mapping relationship between the target data file and the at least one tag.

2. The method according to claim 1, characterized in that: The determining at least one tag corresponding to the target data file includes: Obtaining label data imported corresponding to the target data file; The tag data is parsed to obtain at least one tag as at least one tag corresponding to the target data file.

3. The method according to claim 2, characterized in that The step of acquiring the label data imported corresponding to the target data file includes: Acquire configuration data imported corresponding to the target data file, wherein the configuration data includes label data configured corresponding to the target data file; At least one tag corresponding to the target data file is extracted from the configuration data.

4. The method according to claim 3, characterized in that The step of obtaining the configuration data imported corresponding to the target data file includes: Acquire interactive interface input information corresponding to the target data file, and extract the label data from the interactive interface input information.

5. The method according to claim 2, characterized in that: The step of acquiring the label data imported corresponding to the target data file includes: The tag data corresponding to the target data file is obtained from a third-party system interface.

6. The method according to claim 1, characterized in that The determining at least one tag corresponding to the target data file includes: Parsing the target data file, and extracting at least one tag indication data from the target data file; At least one tag corresponding to the tag indication data is obtained as at least one tag corresponding to the target data file.

7. The method according to claim 1, characterized in that The method further comprises: Compare the newly imported data files with the stored data files; In the case where the newly imported data file is identical to any of the stored data files, at least one label corresponding to the newly imported data file is compared with an existing label of the stored identical data file, a target label different from the existing label in the at least one label of the newly imported data file is obtained, the target label is stored in the at least one label corresponding to the stored identical data file, and the newly imported data file is discarded; When there is no data file identical to the newly imported data file among the stored data files, the newly imported data file and at least one corresponding label are stored.

8. The method according to claim 1, characterized in that The method further comprises: When the trigger condition is met, label identification is performed on at least one stored data file according to the newly added label identification rule to obtain a new label corresponding to each of the at least one data file; Adding the new tag to the existing tags of the corresponding data file in the mapping relationship; The trigger condition includes at least one of the following: reaching a timing moment; or detecting a trigger event.

9. The method according to claim 8, characterized in that The at least one tag belongs to at least one of the following tag types: Basic feature type labels; Business type label; Custom type tags.

10. The method according to claim 1, characterized in that The managing the target data file based on the stored mapping relationship between the target data file and the at least one tag includes: The tag to be queried is obtained, and based on the stored mapping relationship between the target data file and the at least one tag, the target data file corresponding to the tag to be queried is obtained, and relevant operations are performed on the target data file.

11. A data management device, characterized in that: include: An acquisition module is used to acquire a target data file; A processing module, configured to determine at least one tag corresponding to the target data file, and store a mapping relationship between the target data file and the at least one tag; A management module is used to manage the target data file based on the stored mapping relationship between the target data file and the at least one tag.

12. An electronic device, characterized in that: include: at least one processor; A memory having at least one program stored thereon, wherein when the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 10; At least one I / O interface is connected between the processor and the memory and is configured to implement information interaction between the processor and the memory.

13. A computer readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images