Data annotation method and system for integrating unstructured data with data catalogs

Abstract

A method for performing data management, comprising using the processor of an agent server to generate terminology for structured data files using structured data profiling and generating terminology for unstructured data files using unstructured data profiling, wherein the structured and unstructured data files are stored in storage, and the method for managing a terminology list, the terminology list storing and managing terminology generated by the processor, wherein the processor utilizes the terminology generated by structured data profiling when deriving terminology generated by unstructured data profiling.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Application No. 18 / 213,303, filed on June 23, 2023, entitled "DATA ANNOTATION METHOD AND SYSTEM FOR UNSTRUCTURED DATA INTEGRATING WITH DATA CATALOG", which is hereby incorporated by reference in its entirety.

Background Art

[0002] This disclosure generally relates to methods and systems for performing data management.

[0003] A data lake is a related technique for managing data in enterprise systems. A data lake manages multiple types of data, including structured data (tables, CSV, JSON, etc.) and unstructured data (emails, documents, PDFs, videos, etc.). Today, data lakes have become essential for realizing integrated data analysis in enterprises.

[0004] One of the major difficulties in using a data lake lies in "data search", which means it is difficult for users to find appropriate data for their data analysis. To mitigate this difficulty, data catalogs are adopted in data lakes. A data catalog provides a list of data in the data lake and facilitates the identification and exploration of data.

[0005] A data catalog includes components such as a data crawler and a data profiler. The data crawler finds all the data contained in the data lake and develops a list of the data contained in the data lake. On the other hand, the data profiler performs the function of profiling the data contained in the data lake for data search.

[0006] Figure 1 illustrates a data management system employing the data lake methodology of related technologies. A data crawler finds all data within the data lake at each agent. A data profiler extracts characteristics of the identified data. The results of data profiling are provided to a data steward. Using the profiling results, the data steward defines business terms based on the data profile, meaning the steward defines terms common to similar data such as address, vehicle number, name, and age. Using a data catalog, users can not only obtain a list of data in the data lake but also search for data using business terms for their own data analysis. Therefore, the efficiency of data retrieval is improved by utilizing a data catalog.

[0007] Related technologies disclose methods for improving the efficiency of profiling large datasets. This method parses files using several parsers to generate schemas. Document / file formats can be discovered by attempting to parse the files using several parsers. However, this method does not utilize definitions of business terms derived from structured and unstructured data when performing file parsing.

[0008] Related technologies disclose methods for identifying and classifying data using advanced machine learning algorithms. These methods provide a visual representation of categories in data centers and data infrastructure distributed across multiple clusters. However, these related technologies also do not utilize definitions of business terminology to enable data discovery.

[0009] Related technologies disclose methods for implementing cognitive data lakes that select or recommend operational databases based on historically created data lakes. These methods provide selection or recommendations for operational databases based on historically created data lakes that store files with similar file time, classification, metadata, and file usage frequency. However, these methods do not utilize such information to improve the efficiency of business terminology definitions.

[0010] However, the related technologies disclosed above define business terminology separately for structured and unstructured data. As a result, there are various problems and shortcomings. For example, in the related technologies, the definitions of business terminology within structured data are not efficiently utilized in relation to unstructured data, and vice versa. [Overview of the project] [Means for solving the problem]

[0011] The aspects of this disclosure include innovative methods for performing data management. These methods may include using an agent server processor to generate terminology for structured data files using structured data profiling and generating terminology for unstructured data files using unstructured data profiling, wherein the structured and unstructured data files are stored in storage, and managing a terminology list, the terminology list storing terminology generated by the processor, and the processor utilizing and managing the terminology generated by structured data profiling when deriving terminology generated by unstructured data profiling.

[0012] The aspects of this disclosure include an innovative non-temporary computer-readable medium for storing instructions for performing data management. Instructions may include using an agent server processor to generate terminology for structured data files using structured data profiling and generating terminology for unstructured data files using unstructured data profiling, wherein the structured and unstructured data files are stored in storage; generating; and managing a terminology list, which stores the terminology generated by the processor, and which the processor uses the terminology generated by structured data profiling when deriving the terminology generated by unstructured data profiling.

[0013] The aspects of this disclosure include an innovative server system for performing data management. The system uses an agent server processor to generate terminology for structured data files using structured data profiling and to generate terminology for unstructured data files using unstructured data profiling, wherein the structured and unstructured data files are stored in storage, and manages a terminology list, the terminology list stores terminology generated by the processor, and the processor uses and manages the terminology generated by structured data profiling when deriving terminology generated by unstructured data profiling.

[0014] The aspects of this disclosure include an innovative system for performing data management. The system may include means for generating terms for structured data files using an agent server processor and generating terms for unstructured data files using unstructured data profiling, wherein the structured and unstructured data files are stored in storage; and means for managing a term list, wherein the term list stores terms generated by the processor, and the processor utilizes the terms generated by structured data profiling when deriving terms generated by unstructured data profiling.

[0015] The aspects of this disclosure include an innovative server system for performing data management. The system includes at least one agent server, each of which includes at least one processor configured to perform structured data profiling to generate terms for structured data files and to perform unstructured data profiling to generate terms for unstructured data files; at least one storage, each of which is associated with an individual agent server and stores structured data files and unstructured data files; and a management server that communicates with at least one agent server and includes a management processor configured to manage a term list, the term list stores terms generated by at least one processor, and each of the at least one processor is configured to utilize terms generated by performing structured data profiling when deriving terms generated by performing unstructured data profiling. [Brief explanation of the drawing]

[0016] Next, the general architecture for implementing the various features of the present disclosure is described with reference to the drawings. The drawings and the related description are provided to illustrate exemplary implementations of the present disclosure, not to limit the scope of the present disclosure. Throughout the drawings, reference numbers are reused to indicate the correspondence between the elements being referenced.

[0017] [Figure 1] FIG. 1 shows a data management system adopting a data lake approach of the related art.

[0018] [Figure 2] FIG. 2 shows an exemplary block diagram of a data annotation system 200 according to an exemplary implementation.

[0019] [Figure 3] FIG. 3 shows an exemplary agent list 212 according to an exemplary implementation.

[0020] [Figure 4] FIG. 4 shows an exemplary structured data list 213 according to an exemplary implementation.

[0021] [Figure 5] FIG. 5 shows an exemplary structured data list 214 according to an exemplary implementation.

[0022] [Figure 6] FIG. 6 shows an exemplary business term list 215 according to an exemplary implementation.

[0023] [Figure 7] FIG. 7 shows an exemplary business term template 216 according to an exemplary implementation.

[0024] [Figure 8] FIG. 8 shows an exemplary structured data file 801 according to an exemplary implementation.

[0025] [Figure 9] Figure 9 shows an exemplary structured data file 901, which follows an exemplary implementation.

[0026] [Figure 10] Figure 10 shows an exemplary unstructured data file 1001, which follows an exemplary implementation.

[0027] [Figure 11] Figure 11 shows an exemplary unstructured data file 1101, which follows an exemplary implementation configuration.

[0028] [Figure 12] Figure 12 shows an exemplary representation 1200 of the data catalog operator 209 according to an exemplary implementation.

[0029] [Figure 13] Figure 13 shows an exemplary processing flow 1300 of the crawling manager 210, following an exemplary implementation.

[0030] [Figure 14] Figure 14 shows an exemplary processing flow 1400 of the data crawler 221 according to an exemplary implementation.

[0031] [Figure 15] Figure 15 shows an exemplary processing flow 1500 of a structured data profiler 222, following an exemplary implementation.

[0032] [Figure 16] Figure 16 shows an exemplary process flow 1600 of an unstructured data profiler 223, following an exemplary implementation.

[0033] [Figure 17] Figure 17 shows an exemplary processing flow 1700 of the business term configurator 211, following an exemplary implementation.

[0034] [Figure 18] Figure 18 shows an exemplary output display 1800 of the data catalog viewer 208, according to an exemplary implementation configuration.

[0035] [Figure 19] Figure 19 shows an exemplary computing environment with exemplary computer equipment suitable for use in several exemplary implementation forms. [Modes for carrying out the invention]

[0036] The following detailed description provides further details of the drawings and implementation examples of this application. For clarity, redundant reference numbers and descriptions of elements between the drawings have been omitted. Terms used throughout this description are provided as examples and are not intended to be limiting. For example, the use of the term “automatic” may include a fully automatic implementation or a semi-automatic implementation that includes user or administrator control over certain aspects of the implementation, depending on the desired implementation for those skilled in the art practicing the implementation of this application. Selection may be made by the user via a user interface or other input means, or may be implemented by a desired algorithm. The implementation examples described herein may be used individually or in combination, and the functionality of the implementation examples may be implemented by any means depending on the desired implementation.

[0037] Figure 2 shows an exemplary block diagram of a data annotation system 200 following an exemplary implementation. As shown in Figure 2, the data annotation system 200 may include, but is not limited to, components such as a management server 201, an agent server 202, and storage 203. The management server 201 may include components such as a central processing unit (CPU) 204, an internal network 205, memory 206, and a data catalog manager 207. The internal network 205 facilitates communication between the CPU 204 and various components. Memory 206 stores the data catalog manager 207. The data catalog manager 207 may include software components such as a data catalog viewer 208, a data catalog operator 209, a crawling manager 210, a business term configurator 211, an agent list 212, a structured data list 213, an unstructured data list 214, a business term list 215, and a business term template 216.

[0038] The data catalog viewer 208 generates and edits a list of data entries and related information for the user to search and review. The data catalog operator 209 displays suggested business terms for the user to review and select. The crawl manager 210 manages the crawled data. The business term configurator 211 generates suggested business terms.

[0039] The agent list 212 stores agent server information for identifying and searching for each agent server. The structured data list 213 stores information about lists of structured data. The unstructured data list 214 stores information about lists of unstructured data. The business term list 215 stores information about business term identifiers and their associated definitions / descriptions. The business term template 216 stores information about structured templates associated with business terms. The crawling manager 210 uses the agent list 212, structured data list 213, unstructured data list 214, business term list 215, and business term template 216 to perform data crawling.

[0040] The management server 201 communicates with the agent servers 202 via the network 226. Each agent server 202 may include components such as a CPU 217, a bus 218, and memory 219. The CPU 217 communicates with the memory 219 via the bus 218. The memory 219 may include a data catalog agent 220, which may include components such as a data crawler 221, a structured data profiler 222, and an unstructured data profiler 223. Details of the data crawler 221, structured data profiler 222, and unstructured data profiler 223 are described in detail below.

[0041] Each agent server 202 is connected to individual storage 203 that store structured data files 224 and unstructured data files 225. The data crawler 221 searches for data files from the corresponding storage 203 by searching for structured data files 224 and unstructured data files 225. The identified and discovered data files are then data-profiled using either a structured data profiler 222 or an unstructured data profiler 223, depending on the file type.

[0042] Figure 3 shows an exemplary agent list 212 following an exemplary implementation. The agent list 212 may include, but is not limited to, information such as agent ID 301, Internet Protocol (IP) address 302, type 303, and access method 304. The agent ID 301 is the identifier of the agent server 202. The IP address 302 is the network address of the agent server 202. The type 303 indicates the type of data within the agent server 202, such as a file or database. The access method 304 indicates the method of accessing the data.

[0043] Figure 4 shows an exemplary structured data list 213 following an exemplary implementation. The structured data list 213 may include, but is not limited to, information such as structured data ID 401, agent ID 402, table name 403, column name 404, data profile 405, and business term ID 406. The structured data ID 401 is an identifier for the structured data. The agent ID 402 is the agent ID 301 that stores the data. The table name 403 indicates the name of the structured data. The column name 404 indicates the column name in the data. The data profile 405 indicates the characteristics of the data. The business term ID 406 indicates the business term tagged with the data, corresponding to an ID defined in the business term list 215.

[0044] Figure 5 shows an exemplary structured data list 214 following an exemplary implementation. The structured data list 214 may include, but is not limited to, information such as an unstructured data ID 501, an agent ID 502, a file name 503, a business term ID 504, and supplementary information 505. The unstructured data ID 501 is an identifier for the unstructured data. The agent ID 502 is the agent ID 301 that stores the data. The file name 503 indicates the name of the unstructured data file 225 that stores the data. The business term ID 504 indicates the business term tagged with the data, corresponding to the ID defined in the business term list 215. The supplementary information 505 indicates additional information about the unstructured data and business term.

[0045] Figure 6 shows an exemplary business term list 215 following an exemplary implementation. The business term list 215 may include, but is not limited to, information such as business term ID 601 and business term 602. Business term ID 601 is an identifier for a business term. Business term 602 indicates a business term related to business term ID 601.

[0046] Figure 7 shows an exemplary business term template 216 following an exemplary implementation. As shown in Figure 7, the business term template 216 may include, but is not limited to, information such as business term ID 701, business term template 702, and business term 703. Business term ID 701 is an identifier for a business term. Business term template 702 shows a term template indicating the proximity and relationship of words and numbers related to business term template 702. Business term 703 shows a business term associated with business term ID 601. Taking the second entry as an example, business term ID "101" is associated with the business term template "( / d * ) / d * - / d * It has the defined business term "telephone number". This is "( / d * ) / d * - / d *", here " / d * The text string matching the business term template "" represents a number / digit, meaning that it should be associated with the business term "telephone number".

[0047] Figure 8 shows an exemplary structured data file 801 following an exemplary implementation. As shown in Figure 8, structured data file 801 contains column data organized by column names: Name, Address, and Nationality. Figure 9 shows an exemplary structured data file 901 following an exemplary implementation. As shown in Figure 9, structured data file 901 contains column data organized by column names / categories: Name, Address, and Nationality. In contrast to the commas shown in Figure 8, the columns are separated into diagrammed columns.

[0048] Figure 10 shows an exemplary unstructured data file 1001 according to an exemplary implementation. As shown in Figure 10, the unstructured data file 1001 may contain information in a structured format (e.g., classified information), but compared to the examples of structured data files 801 and 901 shown in Figures 8 and 9, the information is not otherwise a structured list of data. Figure 11 shows an exemplary unstructured data file 1101 according to an exemplary implementation. As shown in Figure 11, the unstructured data file may take a less structured form than that of Figure 10. For example, the unstructured data file 1101 represents an email communication that contains information in the body of the communication.

[0049] Figure 12 shows an exemplary display 1200 of the data catalog operator 209, following an exemplary implementation. Display 1200 enumerates the data profiles expanded by the structured data profiler 222 and the unstructured data profiler 223. Display 1200 may also display information including, but is not limited to, a start button 1201, an agent ID 1202, suggested business terms 1203, a table / file name 1204, a column name 1205, supplementary information 1206, and a decision 1207. Clicking the start button 1201 initiates the data crawling process. The agent ID 1202 identifies the agent server that stores the data. Suggested business terms 1203 represent business terms entered as suggestions by the business term configurator 211. Details of the suggested business term generation process are described in detail in Figures 15 and 17 below. Table / filename 1204 is the same as table name 403 in Figure 4 and filename 503 in Figure 5, and is associated with business terms as proposed. Column name 1205 is the same as column name 404 in Figure 4, and is associated with business terms as proposed. Supplementary information 1206 is the same as supplementary information 505 in Figure 5, and is associated with business terms as proposed.

[0050] Decision 1207 may include user-selectable decision buttons such as an Accept button 1208, a Reject button 1209, and a Term Modification button 1210. The user is given the option to accept the generated proposal, reject the proposal, or modify the terms associated with the proposal. If the Accept button 1208 is selected, the user accepts the proposed business terms 1203 associated with the entry, and the business catalog operator 209 updates the business term list 215 to register the accepted business terms.

[0051] If the reject button 1210 is selected, the user rejects the suggested business term 1203 associated with the entry, and the business catalog operator 209 does not update the business term list 215. On the other hand, if the term modification button 1211 is selected, the user needs to modify the business term 1203 associated with the entry. The catalog operator 209 then prompts the user to enter the new business term and updates the business term list 215 to register the business term entered by the user.

[0052] Figure 13 shows an exemplary processing flow 1300 of the crawling manager 210 according to an exemplary implementation. This process begins in step S1302, in which the structured data list 213, the unstructured data list 214, the business term list 215, and the business term template 216 are sent to the data crawler 221. In step S1304, the agent list 212 is received. Next, in step S1306, data crawling is started using the data crawler 221 on the agent servers included in the agent list 212. In step S1308, the crawling results are sent to the business term configurator 211 for processing.

[0053] Figure 14 shows an exemplary processing flow 1400 of the data crawler 221 according to an exemplary implementation. This process begins in step S1402, in which a data file search is performed on the storage 203. In step S1404, it is determined whether the data file is in the structured data list 213 or the unstructured data list 214. If the answer is No, the process proceeds to step S1414, which is described in more detail below. If the answer is Yes, the process continues to step S1406, in which it is determined whether the file is structured data.

[0054] If the answer is Yes, the process proceeds to step S1408. In step S1408, the structured data profiler 222 is started and performs data profiling on the data file. Once data profiling is complete, in step S1412, the profiling results are sent to the crawling manager 210. On the other hand, if the answer in step S1406 is No, the process proceeds to step S1410, in which the unstructured data profiler 223 is started and performs data profiling on the data file. Once data profiling is complete, in step S1412, the profiling results are sent to the crawling manager 210.

[0055] Once step S1412 is complete, step S1414 determines whether all data files stored in storage 203 have been found. If all data files have been found, the process terminates. On the other hand, if not all data files have been found, the process returns to step S1402 and continues until all data files have been found.

[0056] Figure 15 shows an exemplary processing flow 1500 of the structured data profiler 222, following an exemplary implementation. As shown in Figure 15, the process begins in step S1502, where column names are extracted from the structured data file 224. In step S1504, column data is extracted from the structured data file 224. The process then proceeds to step S1506, where it is determined whether the extracted column data matches the business term template 702. Specifically, it is determined whether the extracted column data matches any template that shows the proximity and relationships between words and numbers related to business terms contained within the business term template 702. In step S1508, business term suggestions are formulated / added, which are described in more detail in Figure 17 below.

[0057] Next, in step S1510, data profiling is performed to examine, analyze, and summarize the extracted column data. Once data profiling is complete, in step S1512, the data profile is sent to the data crawler 221. If business term suggestions were performed in S1508, the suggested business terms are also sent to the data crawler 221 in step S1512. In step S1514, it is determined whether all data columns have been extracted. If the answer is Yes, the process stops. If the answer is No, the process proceeds to step S1502 to perform additional extractions.

[0058] Figure 16 shows an exemplary processing flow 1600 of the unstructured data profiler 223 according to an exemplary implementation. In step S1602, it is determined whether the unstructured data file 225 matches the target format of the unstructured data profiler 223. This determination is made by examining the file type from the file extension and determining whether the file type matches one of the two data profilers.

[0059] In step S1604, data profiling is performed on the unstructured data file 225 using terms contained in the business term list 215. In scenarios where the unstructured data file 225 is an image, an image annotator is used to find objects in the business term list 215. If the image annotator finds an object enumerated in the business term list 215 through cross-referencing, the process proceeds to step S1606, in which the found business terms are sent to the data crawler 221. In some exemplary implementations, supplementary information related to the unstructured data file 225 is also generated during step S1604 and sent to the data crawler 221 along with the found business terms in step S1606.

[0060] In a scenario where the unstructured data file 225 is a document containing tables, the document annotator is used to extract data names and values ​​from the tables contained in the document. In step S1604, the document annotator is used to find data in the business term list 215. If the document annotator matches the extracted data with terms in the business term list 215 to find business terms, in S1606 the found business terms are sent to the data crawler 221. In some exemplary implementations, supplementary information related to the unstructured data file 225 is also generated during step S1604 and sent to the data crawler 221 along with the found business terms in step S1606.

[0061] In a scenario where the unstructured data file 225 is a document, a document annotator is used to summarize the document. In step S1604, the document annotator extracts sentences from the summary and is used to match the terms contained in the sentences with terms in the business term list 215. In S1606, the discovered business terms are sent to the data crawler 221. In some exemplary implementations, supplementary information related to the unstructured data file 225 is also generated during step S1604 and sent to the data crawler 221 in step S1606 along with the discovered business terms.

[0062] In some exemplary implementations, supplemental information and terminology generated by unstructured data profiling may be used to further generate additional / subsequent terminology in the structured data file.

[0063] Figure 17 shows an exemplary processing flow 1700 of the business term configurator 211 according to an exemplary implementation. In step S1702, it is determined whether the crawling results generated by the crawling manager 210 relate to structured data or unstructured data. If the crawling results relate to structured data, the process proceeds to step S1704, in which the crawling results are compared for similarity with the data profile 405, which includes business term suggestions formulated / added in step S1508. If the crawling results relate to unstructured data, this process The process continues to step S1706, in which the crawling results are compared with supplemental information 505 for similarity.

[0064] Once step S1704 or S1706 is completed, the process proceeds to step S1708, where it is determined whether the similarity determined in step S1704 or S1706 exceeds a threshold (similarity threshold). If the determined similarity exceeds the threshold in step S1708, relevant business terms are added as suggestions in step S1710. If the determined similarity does not exceed the threshold in step S1708, the process terminates.

[0065] Figure 18 shows an exemplary output display 1800 of the data catalog viewer 208, following an exemplary implementation. As shown in Figure 18, the user may enter search keywords in the search box 1801 to search for data entries. The data catalog viewer 208 then searches the business term list 215, the structured data list 213, and the unstructured data list 214 in relation to the search keywords. The output display 1800 then displays information including the agent ID 1802, suggested business terms 1803, table / file name 1804, column name 1805, supplementary information 1806, and file type 1807.

[0066] The exemplary implementations described above may have various benefits and advantages. For example, improved efficiency in developing business terminology for unstructured data files using business terminology developed for structured data files, and vice versa. In addition, the usefulness and accuracy of business terminology within the data lake are improved by linking business terminology developed for both structured and unstructured data.

[0067] Figure 19 shows an exemplary computing environment having exemplary computer equipment suitable for use in several exemplary implementations. The computer equipment 1905 within the computing environment 1900 may include one or more processing units, cores, or processors 1910, memory 1915 (e.g., RAM, ROM, and / or similar), internal storage 1920 (e.g., magnetic, optical, solid-state storage, and / or organic) and / or I / O interfaces 1925, any of which may be connected on a communication mechanism or bus 1930 for transmitting information, or may be incorporated within the computer equipment 1905. The I / O interface 1925 may also be configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0068] Computer device 1905 may be communicatively connected to input / user interface 1935 and output device / interface 1940. One or both of input / user interface 1935 and output device / interface 1940 may be wired or wireless interfaces and may be detachable. Input / user interface 1935 may include any physical or virtual device, component, sensor, or interface that can be used to provide input (e.g., buttons, touchscreen interfaces, keyboards, pointing / cursor controls, microphones, cameras, Brailles, motion sensors, accelerometers, optical readers, and / or similar). Output device / interface 1940 may include displays, televisions, monitors, printers, speakers, Brailles, or similar. In some exemplary implementations, input / user interface 1935 and output device / interface 1940 may be integrated with or physically connected to computer device 1105. In other exemplary implementations, other computer devices may function as or provide input / user interfaces 1935 and output devices / interfaces 1940 for computer device 1905.

[0069] Examples of computer devices 1905 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles or other machines, devices carried by humans and animals, and similar devices), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, and similar devices), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions, radios, and similar devices having one or more processors built in and / or connected thereto).

[0070] Computer device 1905 may be communicably connected (for example, via I / O interface 1925) to external storage 1945 and network 1950 for communication with any number of networked components, devices, and systems, including one or more computer devices of the same or different configurations. Computer device 1905 or any connected computer device may function as, provide, or be referred to as a server, client, thin server, general-purpose machine, special-purpose machine, or other label.

[0071] The IO interface 1925 may include, but is not limited to, wired and / or wireless interfaces that use any communication or IO protocol or standard (e.g., Ethernet, 902.11x, Universal Serial Bus, WiMAX, modem, cellular network protocol, and similar) to communicate information to and from at least all connected components, devices, and networks within the computing environment 1900. The network 1950 may be any network or combination of networks (e.g., the Internet, local area network, wide area network, telephone network, cellular network, satellite network, and similar).

[0072] Computer devices 1905 may use and / or communicate using computer-usable or computer-readable media, including temporary and non-temporary media. Temporary media include transmitting media (e.g., metal cables, optical fibers), signals, carriers, and similar media. Non-temporary media include magnetic media (e.g., disks and tapes), optical media (e.g., CD-ROMs, digital video discs, Blu-ray discs), solid-state media (e.g., RAM, ROMs, flash memory, solid-state storage), and other non-volatile storage or memory.

[0073] Computer devices 1905 may be used to implement techniques, methods, applications, processes, or computer executable instructions in several exemplary computing environments. Computer executable instructions may be obtained from temporary media and stored on and retrieved from non-temporary media. Executable instructions may originate from one or more arbitrary programming, scripting, and machine languages ​​(e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).

[0074] Processor 1910 may run under any operating system (OS) (not shown) in a native or virtual environment. One or more applications may be deployed with the OS and other applications (not shown), including a logical unit 1960, an application programming interface (API) unit 1965, an input unit 1970, an output unit 1975, and an inter-unit communication mechanism 1996 for different units to communicate with each other. The units and elements described may be modified in design, function, configuration, or implementation, and are not limited to the description provided. Processor 1910 may be in the form of a hardware processor, such as a central processing unit (CPU), or a combination of hardware and software units.

[0075] In some exemplary implementations, information or execution instructions, upon being received by the API unit 1965, may be transmitted to one or more other units (e.g., a logical unit 1960, an input unit 1970, and an output unit 1975). In some cases, the logical unit 1960 may be configured to control the flow of information between units and to control the services provided by the API unit 1965, the input unit 1970, and the output unit 1975 in some exemplary implementations described above. For example, the flow of one or more processes or implementations may be controlled by the logical unit 1960 alone or in conjunction with the API unit 1965. The input unit 1970 may be configured to take input for a computation described in an exemplary implementation, and the output unit 1975 may be configured to provide output based on a computation described in an exemplary implementation.

[0076] The processor 1910 may be configured to generate terminology for structured data files using structured data profiling and terminology for unstructured data files using unstructured data profiling, and the structured and unstructured data files are stored in storage as shown in Figure 14. The processor 1910 may also be configured to manage a terminology list, which stores the generated terminology as shown in Figure 14. The processor 1910 may also be configured to utilize the terminology generated by structured data profiling when deriving terminology generated by unstructured data profiling. The processor 1910 may also be configured to manage a terminology list, which stores the generated terminology as shown in Figures 1 and 14.

[0077] The processor 1910 may be configured to search for files stored in storage that are used by the processor for term generation, as shown in Figure 1. The processor 1910 may be configured to generate supplemental information related to terms generated by structured data profiling and unstructured data profiling, as shown in Figures 5 and 12. For unstructured data files containing images, the processor 1910 may be configured to perform image annotation on the images to identify objects from the images, cross-reference the identified objects against terms in the term list to find matching terms, and output the matching terms as generated terms for the unstructured data file containing images, as shown in Figure 12. The processor 1910 may be configured to display the terms generated for structured data files and the terms generated for unstructured data files, as shown in Figures 12 and 17, so that the user can accept, reject, or modify the terms generated by structured data profiling and unstructured data profiling.

[0078] Some parts of the detailed explanation have been presented concerning algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are means used by those skilled in data processing technology to convey the essence of the innovation to others skilled in the art. An algorithm is a set of predefined steps that produce a desired end state or result. In one implementation example, the steps performed require the physical manipulation of tangible quantities to achieve a specific result.

[0079] Unless otherwise specified, explanations that use terms such as “processing,” “computing,” “calculating,” “determining,” and “displaying” throughout the explanation, as is evident from the explanation, should be understood to include actions and processes of a computer system or other information processing device that manipulate data represented as physical (electronic) quantities in the registers and memory of a computer system and convert it into other data similarly represented as physical quantities in the memory or registers of a computer system, or in other information storage, transmission, or display devices.

[0080] Implementation examples may also relate to apparatus for performing the operations described herein. Such apparatus may be specifically constructed for a required purpose and may include one or more general-purpose computers that are selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored in computer-readable media such as computer-readable storage media or computer-readable signal media. Computer-readable storage media may include, but are not limited to, tangible media such as optical disks, magnetic disks, read-only memory, random-access memory, solid-state devices and drives, or any other type of tangible or non-temporary media suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and representations presented herein are not specific to any particular computer or other apparatus. Computer programs may include pure software implementations containing instructions for performing the operations in a desired implementation form.

[0081] Various general-purpose systems may be used with the programs and modules illustrated herein, or it may be convenient to construct more specialized devices for performing desired method steps. Furthermore, the implementation examples do not describe any particular programming language. It will be understood that various programming languages ​​may be used to implement the teachings of the implementation examples described herein. Instructions in a programming language may be executed by one or more processing units, such as a central processing unit (CPU), processor, or controller.

[0082] As is known in the art, the operations described above can be performed by hardware, software, or any combination of software and hardware. Various aspects of the implementation examples may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions (software) stored on a machine-readable medium, which, when executed by a processor, cause the processor to execute a method for performing the implementation of the present application. Furthermore, some implementation examples of the present application may be performed by hardware alone, while others may be performed by software alone. Moreover, the various functions described may be performed within a single unit or distributed across several components in any number of ways. When performed by software, the method may be executed by a processor such as a general-purpose computer based on instructions stored on a computer-readable medium. If desired, the instructions may be stored on the medium in compressed and / or encrypted form.

[0083] Furthermore, other implementations of the Application will become apparent to those skilled in the art by examining this Specification and practicing the Techniques of the Application. Various aspects and / or components of the implementations described herein may be used individually or in any combination. This Specification and the implementations are intended to be considered merely as examples, and the true scope and spirit of the Application are indicated by the appended claims.

Claims

1. A data management system, The aforementioned system, At least one agent server, and each at least one agent server is Perform structured data profiling to generate terminology for structured data files, and An agent server comprising at least one processor configured to perform unstructured data profiling to generate terminology for unstructured data files, At least one storage, each at least one storage associated with an individual agent server, storing the structured data files and the unstructured data files, A management server that communicates with the at least one agent server, wherein the management server A management server includes a management processor configured to manage a term list, the term list storing terms generated by at least one processor, and the management server includes a management processor. Includes, A data management system in which each of the at least one processor is configured to utilize terms generated by performing structured data profiling when deriving terms generated by performing unstructured data profiling.

2. The system according to claim 1, wherein the at least one processor is further configured to perform data crawling, the data crawling retrieving files stored in associated storage for use by the associated processor for term generation.

3. The system according to claim 1, wherein the at least one processor is further configured to generate terms generated by unstructured data profiling and supplemental information relating to the unstructured data file.

4. The system according to claim 3, wherein the at least one processor is further configured to generate subsequent terms for the structured data file using the terms and supplementary information generated by unstructured data profiling.

5. The system according to claim 3, wherein the at least one processor is further configured to generate supplemental information by performing sentence extraction on the unstructured data file based on terms stored in the term list.

6. For an unstructured data file containing an image, the associated processor of the at least one processor performs image annotation on the image to identify objects from the image, cross-references the identified objects with terms in the term list to find matching terms, and outputs the matching terms as generated terms for the unstructured data file containing the image. The system according to claim 3.

7. The system according to claim 1, wherein the terms generated by structured data profiling and unstructured data profiling are displayed for the user to accept, reject, or modify each of the terms generated for the structured data file and each of the terms generated for the unstructured data file.

8. A method for performing data management, The process involves using the agent server's processor to generate terminology for structured data files using structured data profiling and generating terminology for unstructured data files using unstructured data profiling, wherein the structured data files and the unstructured data files are stored in storage. Managing a term list, wherein the term list stores and manages terms generated by the processor. Includes, The processor utilizes terms generated by structured data profiling when deriving terms generated by unstructured data profiling. method.

9. The processor searches for files stored in the storage used by the processor for term generation. The method according to claim 8, further comprising:

10. The processor generates the terms generated by unstructured data profiling and supplementary information related to the unstructured data file. The method according to claim 8, further comprising:

11. The processor generates subsequent terms for the structured data file by using the supplemental information and terms generated by unstructured data profiling. The method according to claim 10, further comprising:

12. The generation of the aforementioned supplementary information is The processor generates the supplementary information by performing sentence extraction on the unstructured data file based on the terms stored in the term list. The method according to claim 10, including the method described in claim 10.

13. For an unstructured data file containing images, the processor performs image annotation on the images to identify objects from the images, cross-references the identified objects with terms in the term list to find matching terms, and outputs the matching terms as generated terms for the unstructured data file containing the images. The method according to claim 10, further comprising:

14. To display each of the generated terms for the structured data file and each of the generated terms for the unstructured data file, the terms generated by structured data profiling and unstructured data profiling, so that the user can accept, reject, or modify them. The method according to claim 8, further comprising:

15. A non-temporary computer-readable medium for storing instructions for performing data management, wherein the instructions are: The method involves generating terminology for structured data files using structured data profiling and generating terminology for unstructured data files using unstructured data profiling, wherein the structured data files and the unstructured data files are stored in storage, and The management of a term list, the term list storing and managing generated terms. This includes, and the generated terms are used by structured data profiling when deriving terms generated by unstructured data profiling. Non-temporary computer-readable media.

16. Searching for files stored in the aforementioned storage for use in term generation. A non-temporary computer-readable medium according to claim 15, further comprising:

17. To generate the aforementioned terms and supplementary information related to the unstructured data files generated by unstructured data profiling. A non-temporary computer-readable medium according to claim 15, further comprising:

18. To generate subsequent terms for the structured data file by using the supplemental information and terminology generated by unstructured data profiling. A non-temporary computer-readable medium according to claim 17, further comprising:

19. Generating the aforementioned supplementary information means The supplementary information is generated by performing sentence extraction on the unstructured data file based on the terms stored in the aforementioned term list. A non-temporary computer-readable medium according to claim 17, including the following:

20. For an unstructured data file containing images, the process involves performing image annotation on the images to identify objects from the images, cross-referencing the identified objects with terms in a term list to find matching terms, and outputting the matching terms as generated terms for the unstructured data file containing the images. A non-temporary computer-readable medium according to claim 17, further comprising:

Images