Extracting structured information from unstructured documents

Abstract

Embodiments of the present invention provide methods, computer program products, and systems. Embodiments of the present invention can extract structured information for unstructured document analysis. Embodiments of the present invention can extract structured information for unstructured document analysis by identifying tables and columns in a database corresponding to business terms of a business glossary. Then, embodiments of the present invention can receive a designation of a business term of interest identified in an unstructured document. Then, embodiments of the present invention can generate an analysis module based on the identified tables and columns, which enables the identification or recognition of attribute values ​​of the attributes of the tables and columns. Then, embodiments of the present invention can use the analysis module to automatically extract the values ​​of at least some attributes from the unstructured document based on the designation of the business term of interest.

Description

Technical Field

[0001] The present invention relates to the field of digital computer systems, and more particularly, to a method for extracting structured information from unstructured documents. Background Art

[0002] The amount of unstructured documents used for data analysis is growing exponentially. However, unstructured documents may not be queried in a simple way, which significantly limits the extraction of knowledge contained in such documents. Summary of the invention

[0003] Various embodiments provide a method, a computer system and a computer program product for extracting structured information from an unstructured document as described in the subject matter of the independent claims. Advantageous embodiments are described in the dependent claims. If the embodiments of the invention are not mutually exclusive, they can be freely combined with each other.

[0004] In one aspect, the present invention relates to a computer-implemented method for extracting structured information for analysis of unstructured documents. The method includes: identifying tables and columns in a database corresponding to business terms of a business glossary; receiving a specification of a business term of interest for identification in an unstructured document; generating an analysis module based on the identified tables and columns, the analysis module enabling identification or recognition of attribute values ​​of attributes of the tables and columns; and using the analysis module to automatically extract / detect values ​​of at least a portion of the attributes from the unstructured document based on the specification of the business term of interest.

[0005] In another aspect, the present invention relates to a computer program product comprising a computer-readable storage medium having a computer-readable program code, the computer-readable program code being configured to implement all the steps of the method according to the aforementioned embodiments.

[0006] In another aspect, the present invention relates to a computer system configured to: identify tables and columns in a database corresponding to business terms of a business glossary; receive a specification of a business term of interest for identification in an unstructured document; generate an analysis module based on the identified tables and columns, the analysis module enabling identification or recognition of attribute values ​​of attributes of the tables and columns; and based on the specification of the business term of interest, use the analysis module to automatically extract / detect values ​​of at least a portion of the attributes from the unstructured document.

[0007] The present subject matter can enable the use of a computer-implemented method to extract structured information from unstructured documents. This can enable the automatic discovery of relevant information from unstructured documents as structured information. This can make the structured information available to users such as data scientists in a timely manner. The present subject matter can save resources that would otherwise be required to perform a special extraction of structured information from unstructured documents. This can be particularly advantageous as the number of unstructured documents to be analyzed continues to increase. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Embodiments of the present invention are explained in more detail below, by way of example only, with reference to the accompanying drawings, in which:

[0009] Figure 1 is a block diagram of a computer system according to an embodiment of the present invention.

[0010] Figure 2 is a flowchart of a method for extracting structured information from an unstructured document according to an embodiment of the present invention.

[0011] Figure 3 is a flowchart of a method for extracting structured information from an unstructured document according to an embodiment of the present invention.

[0012] Figure 4 is a flowchart of a method for extracting structured information from an unstructured document according to an embodiment of the present invention.

[0013] Figure 5 A computerized system suitable for implementing one or more method steps according to an embodiment of the present invention is represented. DETAILED DESCRIPTION

[0014] The description of various embodiments of the present invention will be presented for the purpose of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or improvements over existing technologies on the market, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

[0015] A business glossary can include a list of business terms with definitions. A business glossary defines terms for an entire business domain. For example, a business glossary defines business concepts for an organization or industry. A business glossary can enable sharing of internal vocabularies within an organization.

[0016] Compared with structured documents, unstructured documents can include unstructured information that does not have a predefined data model or is not organized in a predefined manner. Compared with data that is stored in a database in the form of fields or annotated in a document of a structured document, this may make it difficult to use a program to understand such a document. Unstructured documents can be, for example, electronic documents. Electronic documents can be electronic media content intended to be used in electronic form or as a printout. Electronic documents can include, for example, web pages, documents embedded in web pages and that can be presented in web pages, electronic forms, emails, books, pictures, and presentations, which have associated user agents, such as document readers, editors, or media players.

[0017] The analysis module may be, for example, a software module. The analysis module may include corresponding logic or software for each attribute of the identified table and / or column, which logic or software enables the identification of a value as whether it is a value of the attribute. The identified table and / or column may include one or more record sets, wherein each record set in the record set represents a corresponding different entity type, for example, one record set may be associated with a company, another record set may be associated with a customer, etc. The analysis module may be configured, for example, to determine the entity type represented by a given attribute value in an unstructured document. The extracted values ​​of at least some of the attributes may be provided as structured information by organizing the extracted values ​​of at least some of the attributes into records associated with the corresponding entity type.

[0018] Identifying tables and / or columns in a database corresponding to business terms of a business term table includes, for each business term of the business term table, identifying at least one column and / or at least one table corresponding to the business term. For example, if the business term is "address" and the database includes a table "address" consisting of "street", "zip code", and "city" columns, the entire table may be identified as corresponding to the business term "address". In this case, the attributes "street", "zip code", and "city" are the identified attributes corresponding to the business term "address". In another example, if the business term is "company" and the database includes a table "employee" consisting of "name", "age", and "hiring company" columns, the column "hiring company" may be identified as corresponding to the business term "company". If the database also includes a table named "company" having columns "company name", "location", etc., the column "company name" may be further associated with the business term "company". In this case, two columns are identified as being associated with the business term "company", and the identified attributes corresponding to the business term "company" are the attributes "hiring company" and "company name". The values ​​of the two attributes "hiring company" and "company name" may be (together) used to generate an analysis module so that it may determine whether a value is an attribute value of at least one of the "hiring company" and "company name" attributes. Thus, identifying a table and / or column in a database corresponding to a business term of a business term table includes identifying an attribute in the database corresponding to the business term.

[0019] According to one embodiment, identifying the table and / or column includes: for each term in the business terminology, determining recognition logic based on the format and / or content of the business terminology, and running the recognition logic on the database to identify the table and / or column. The recognition logic may, for example, include a regular expression that can be used to detect a string as a product identifier.

[0020] According to one embodiment, generating the analysis module includes using the attribute values ​​of the identified table and / or column to build a dictionary of business terms, wherein using the analysis module to extract structured information includes comparing the content of the unstructured document with the dictionary. The dictionary can, for example, provide attribute values ​​associated with each term of the business term list as detailed information of the term.

[0021] According to one embodiment, generating the analysis module includes building logic based on the content and / or format of the attribute values ​​of the identified table and / or column so that the logic can identify values ​​similar to the attribute values. The analysis module includes the logic. The analysis module can be generated automatically, for example.

[0022] For example, data profiling of attribute values ​​of each attribute type of the identified table and / or column can be performed. Data profiling can, for example, include format analysis and / or data characteristic analysis. Format analysis of values ​​of an attribute type can create a format expression for values ​​of that attribute type. The format expression can be a pattern containing character symbols for each different character in the column. Data characteristic analysis can determine data characteristics of the attribute value. Data characteristics define characteristics of the data, such as field length or data type. The results of the data profiling can be used to generate logic, for example, a regular expression can be constructed based on the results of the data profiling.

[0023] In one example, the analysis module may include a dictionary and logic, and both (the dictionary and the logic) may be used to automatically extract values ​​of attributes of identified tables and / or columns as structured information from an unstructured document.

[0024] According to one embodiment, the method further comprises: updating the analysis module based on one or more changes in the database and / or the business glossary, and repeating the method using the updated module to extract structured information from the unstructured document and / or from another unstructured document. For example, the update is automatically performed in response to detecting a change. Data changes frequently, and therefore, creating and keeping analysis entities up to date for unstructured content may be technically challenging. This embodiment may provide automatic updates to automatically generated analysis modules.

[0025] According to one embodiment, if the number of changes is above a threshold, an update is performed. For example, if the number of changes is above a threshold, an update may be performed automatically.

[0026] According to one embodiment, extracting structured information includes: identifying values ​​of attributes in the unstructured document corresponding to the attribute values ​​of the identified table and / or column, and forming records associated with the entities from the values ​​according to the entities of the identified table and / or column. For example, the extracted information can be provided as a table or a relational table.

[0027] According to one embodiment, the method further comprises repeating the method for another unstructured document, wherein identification of tables and / or columns is performed in the database and in the formed record.This embodiment may enable a self-improving system based on previously processed unstructured documents.

[0028] According to one embodiment, the analysis module can be a plug-in. The plug-in can be a software component that adds specific features to an existing computer program. This can enable customization of existing programs with the present subject matter.

[0029] According to one embodiment, the database is a master data management (MDM) database. This can achieve seamless integration of the subject matter with existing systems (eg, utilizing their databases).

[0030] Figure 1 An exemplary computer system 100 is depicted. Computer system 100 may be configured, for example, to perform master data management and / or data warehousing. Computer system 100 includes a data integration system 101 and one or more client systems 105 or data sources 106. Client system 105 may include a computer system. Client system 105 may communicate with data integration system 101 via a network connection, including, for example, a wireless local area network (WLAN) connection, a WAN (wide area network) connection, a LAN (local area network) connection, the Internet, or a combination thereof. Data integration system 101 may control access (read and write access, etc.) to a central repository 103 or database.

[0031] The data records stored in the central repository 103 may have values ​​for a set of attributes 109A-P, such as a company name attribute. Although this example is described in terms of a few attributes, more or fewer attributes may be used.

[0032] The data records stored in the central repository 103 may be received from the client system 105 and processed by the data integration system 101 before being stored in the central repository 103. The received records may or may not have the same set of attributes 109A-P. For example, the data record received by the data integration system 101 from the client system 105 may not have all values ​​of the set of attributes 109A-P, for example, the data record may have values ​​of a subset of the attributes of the set of attributes 109A-P, and may not have values ​​of the remaining attributes. In other words, the records provided by the client system 105 may have different completeness. Completeness is the ratio of the number of attributes of the data record that include data values ​​to the total number of attributes in the set of attributes 109A-P. In addition, the records received from the client system 105 may have a different structure than the structure of the stored records of the central repository 103. For example, the client system 105 may be configured to provide records in XML format, JSON format, or other formats that can associate attributes with corresponding attribute values.

[0033] In another example, the data integration system 101 can import data records of the central repository 103 from the client systems 105 using one or more ETL batch processes or via Hypertext Transfer Protocol (HTTP) communications or via other types of data exchange.

[0034] Data integration system 101 can be configured to receive a request from user 110 to perform a particular analysis on an unstructured document. The request can, for example, specify a business term that user 110 is interested in. For example, data integration system 101 can use algorithm 120 to process stored data records 107 in accordance with the present subject matter.

[0035] Figure 2 is a flow chart of a method for extracting structured information from an unstructured document according to an example of the present subject matter. For the purpose of explanation, Figure 2 The method described in Figure 1 The system is implemented in the illustrated system, but is not limited to this implementation. Figure 2 The method can be performed, for example, by the data integration system 101.

[0036] In step 201, a business glossary may be provided. A business glossary may be applicable to data management. A business glossary may include a list of business terms with definitions. A business glossary defines business concepts for an organization or industry. A business glossary may enable sharing of internal vocabularies within an organization.

[0037] In step 203, storage records 107 (e.g., tables and / or columns) corresponding to business terms of the business term list in a database (such as a central repository 103) may be identified. Identifying the tables and / or columns results in identifying records of the tables and / or columns. Identifying the tables and / or columns associated with each business term may include mapping the business term to the tables and / or columns. For example, each term of the business term list may be mapped to a corresponding table and / or column in the database. The mapping may be performed, for example, using software (such as IBM Cloud Pak for Data). The records associated with the tables and columns may be the identified records of step 203. Each identified record may be associated with a corresponding entity. For example, for a specified term such as "address", a table named "address" may be identified, for example, consisting of columns including "street", "zip code", and "city" columns. All records of the identified table "address" may be the identified records of step 203 because the entire table is related to address-related features. Each record of the identified table may have a set of one or more attributes (such as street, zip code, city, etc.) values. Each record of the table may be associated with a corresponding entity (which is an address entity type). In this example, step 203 may result in identifying the attributes "street", "zip code", and "city" as being associated with the business term "address". In another example, for a specified term such as "startup", a column or attribute named "hiring company" in the central repository 103 may be mapped to the term. The column may belong to a table such as named "employee". The table "employee" may include additional attributes, such as the name of the individual, the location of the individual, etc. Each record of the table "employee" may be associated with a corresponding entity (which is an individual). In this case, all records of the table "employee" may be the identified records in step 203, wherein each of all records may include an attribute value, which is the value of the attribute "hiring company". That is, the identified record of step 203 may have the value of the attribute "hiring company" of the corresponding record in the table "employee". In this example, step 203 may result in identifying the attribute "hiring company" as being associated with the business term "company." If the database also includes a table named "company" with columns "company name," "location," etc., the column "company name" may be further associated with the business term "company." Each record of the table "company" may be associated with a corresponding entity, which is a company.

[0038] However, customers may be interested in specific information related to them, such as product names, customer names, employee names, etc. Therefore, in step 205, a designation of a business term of interest for identification in an unstructured document may be received. The designation of a business term may be, for example, a request for a business term received from user 110. For example, user 110 may be interested in a company that has been documented in the form of a book or other unstructured document. The designated business term may be, for example, a term from a business glossary. For example, the designation of a business term in an unstructured document may be received in response to loading the unstructured document into a management database. This may, for example, make the data available to scientists in a timely manner.

[0039] In step 207, an analysis module can be generated based on the attributes of the identified table and / or column. The generated analysis module can make it possible to identify or recognize the attribute values ​​of the identified record. For example, for each attribute type of the identified table and / or column, the analysis module may include logic or data classes that enable the value of the attribute type to be identified. The logic may be, for example, a piece of code, which includes, for example, a regular expression. The analysis module may be configured to read an input value and determine whether the input value is a value of one of the attribute types of the identified record. After the example in which the identified attribute is "hiring company", an analysis module may be generated so that it can determine whether the value is the value of the attribute "hiring company". To this end, the value of the identified column "hiring company" can be used to generate the module. If the database further includes a table named "company", the values ​​of the identified columns "hiring company" and / or "company name" may be used (analyzed) to generate the module.

[0040] The analysis module can be automatically or semi-automatically generated. In a first module generation example, data analysis of the attribute value of each attribute type of the identified table / or column can be performed. In one example, analysis can be performed for the values ​​of more than one attribute type that have been identified as being associated with the same business term in step 203. Data analysis can, for example, include format analysis. The format analysis of the value of the attribute type can create a format expression for the value of the attribute type. The format expression can be a pattern containing character symbols for each different character in the column. For example, each alphabetic character can have a character symbol A, and a numeric character can have a character symbol 9, and the format expression can be used to generate logic for identifying such a pattern, for example, the logic can be configured to map the pattern with an input value. In a second module generation example, the user can be prompted with the values ​​of one or more attribute types of the identified table / or column, or the data analysis results of the value can be prompted to the user, and in response, the defined logic can be received from the user, wherein each defined logic can be configured to identify or recognize the value corresponding to the corresponding attribute type. Therefore, the analysis module can be generated according to the first module generation example and / or the second module generation example.

[0041] In one example, generation of the analysis module may be performed after receiving the designation at step 205. This may be advantageous because it may provide analysis modules on demand. For example, an analysis module may be generated based only on the attribute types of the identified tables and / or columns associated with the specified business terms. This may save resources that would otherwise be required to generate modules for all attribute types. In another example, generation of the analysis module may be performed in advance, such as before step 205. This may prevent the creation of a module for each received request, for example, a single generated module may be used for multiple received designations, such as the received designation of step 205. For example, after the analysis module is generated, steps 205 and 209 may be repeated one or more times using the same generated analysis module for extracting structured information from the same or different unstructured documents.

[0042] The analysis module can be used in step 209 to detect and extract information from the unstructured document based on the designation of the business term of interest. The detected and extracted information can be a value of an attribute type whose value is identified by the analysis module. The detected and extracted information can be referred to as structured information. The detected and extracted information can be provided to the user in a structured format such as a table. The extracted information can include attribute values, each of which is associated with one or more entity types. For example, in the case where the requested business term is about a company, the value identified as the attribute "employment company" or "company name" in the unstructured document can be associated with the entity "person" and the "company" entity. Step 209 can be performed, for example, automatically after receiving the designation and generating the module in step 205. For example, the unstructured document can be parsed, and each parsed value can be processed by the analysis module to determine whether the value is a value of one of the attribute types of the identified table / or column. This step can, for example, result in the identification of multiple values ​​of different attribute types. Each of these multiple values ​​can represent a corresponding one or more entities. For example, if a user requests information about a company, the analysis module may search for a value corresponding to the attribute type "hiring company" of the table because the analysis module is generated based on the value of the "hiring company" attribute.

[0043] Figure 3 is a flow chart of a method for extracting structured information from an unstructured document according to an example of the present subject matter. For the purpose of explanation, Figure 3 The method described in Figure 1 The system is implemented in the illustrated system, but is not limited to this implementation. Figure 3 The method can be performed, for example, by the data integration system 101.

[0044] In step 301, it can be determined whether the number of changes in the central repository 103 exceeds a predefined threshold. The change can be caused by an update and / or insert operation, for example. In the case where the number of changes in the central repository 103 does not exceed the predefined threshold, step 301 can be repeated until the number of changes in the central repository 103 exceeds the predefined threshold, or until the number of repetitions reaches a maximum number of repetitions, so if the maximum number of repetitions is reached, the method can end. In the case where the number of changes in the central repository 103 exceeds the predefined threshold, the changed central repository 103 can be used in step 303 to continuously update the analysis module. The update of the analysis module can be performed by creating new logic and / or updating the existing logic of the analysis module using updated data. The update of the analysis module can be performed using at least one of the first and second module generation examples. In step 305, a designation of a business term of interest identified in an unstructured document can be received. For example, a user can be interested in a company that has been documented in the form of a book or other unstructured document. The designated business term can be, for example, a term of a business glossary. The updated analysis module can be in step 307 (for example, as shown in reference Figure 2 ) as described in step 209 of is used to extract structured information from unstructured documents based on the designation of business terms of interest.

[0045] Figure 4 is a flow chart of a method for extracting structured information from an unstructured document according to an example of the present subject matter. For the purpose of explanation, Figure 4 The method described in Figure 1 The invention is implemented in the system shown in the figure, but is not limited to this implementation. Figure 4 The method can be performed, for example, by the data integration system 101.

[0046] Figure 4 Steps 401 to 409 are Figure 2 Steps 201 to 209. In addition, Figure 4 The method includes repetition of steps 401 to 409, wherein in each repetition, step 403 identifies tables and / or columns of both the database and the structured information extracted in the previous execution of step 409. The repetition of steps 401 to 409 may be performed periodically (e.g., every day), for example. In another example, the repetition of steps 401 to 409 may be performed until a predefined maximum number of repetitions is reached. Figure 4 The method may enable a self-improving system that improves over time using both databases and unstructured documents.

[0047] Figure 5A general purpose computerized system 600 suitable for implementing at least a portion of the method steps as involved in the present disclosure is shown.

[0048] It will be appreciated that the methods described herein are at least partially non-interactive and automated by a computerized system such as a server or embedded system. However, in an exemplary embodiment, the methods described herein may be implemented in (partial) interactive systems. These methods may further be implemented in software 612, 622 (including firmware), hardware (processor) 605, or a combination thereof. In an exemplary embodiment, the methods described herein are implemented as executable programs in software and are performed by a dedicated or general-purpose digital computer (such as a personal computer, workstation, minicomputer, or mainframe computer). Therefore, the most general system 600 includes a general-purpose computer 601.

[0049] In an exemplary embodiment, according to the hardware architecture, Figure 5 As shown, computer 601 includes processor 605, memory (main memory) 610 coupled to memory controller 615 and one or more input and / or output (I / O) devices 10 (or peripheral devices) 645 coupled via local input / output controller 635 communication. Input / output controller 635 can be but not limited to one or more buses or other wired or wireless connections, as known in the art. Input / output controller 635 can have additional elements (omitted for simplicity, such as controller, buffer (cache), driver, repeater and receiver) to realize communication. Further, local interface can include address, control and / or data connection, so that appropriate communication can be realized between the above-mentioned components. As described herein, I / O devices 10, 645 can generally include any universal encryption card or smart card known in the art.

[0050] Processor 605 is a hardware device for executing software, particularly that stored in memory 610. Processor 605 may be any custom or commercially available processor, a central processing unit (CPU), a secondary processor among several processors associated with computer 601, a semiconductor-based microprocessor (in the form of a microchip or chipset), or generally any device for executing software instructions.

[0051] The memory 610 may include any one or combination of volatile storage elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.) and non-volatile storage elements (e.g., ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM)). Note that the memory 610 may have a distributed architecture in which various components are remote from each other but can be accessed by the processor 605.

[0052] The software in the memory 610 may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions, particularly the functions involved in the embodiments of the present invention. Figure 5 In the example of , the software in memory 610 includes instructions 612, such as instructions for managing a database (such as a database management system).

[0053] The software in the memory 610 will typically also include a suitable operating system (OS) 611. The OS 611 essentially controls the execution of other computer programs, such as possible instructions 612 (eg, software) for implementing the methods described herein.

[0054] The methods described herein may take the form of a source program, an executable program (object code), a script, or any other entity including a set of instructions to be executed 612. When a source program, the program needs to be translated by a compiler, an assembler, an interpreter (which may or may not be included in the memory 610), etc., in order to operate correctly with the OS 611. In addition, the methods may be written in an object-oriented programming language with data classes and method classes, or a procedural programming language with routines, subroutines, and / or functions.

[0055] In an exemplary embodiment, a conventional keyboard 650 and mouse 655 may be coupled to the input / output controller 635. Other output devices such as I / O devices 645 may include input devices such as, but not limited to, printers, scanners, microphones, etc. Finally, the I / O devices 10, 645 may further include devices that communicate with both input and output, such as, but not limited to, network interface cards (NICs) or modulators / demodulators (for accessing other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc. The I / O devices 10, 645 may be any common encryption card or smart card known in the art. The system 600 may also include a display controller 625 coupled to the display 630. In an exemplary embodiment, the system 600 may also include a network interface for coupling to a network 665. The network 665 may be an IP-based network for communication between the computer 601 and any external servers, clients, etc. via a broadband connection. The network 665 sends and receives data between the computer 601 and the external system 30, and it may be involved to perform some or all steps of the method discussed herein. In an exemplary embodiment, the network 665 may be a managed IP network managed by a service provider. The network 665 may be implemented wirelessly, for example using wireless protocols and technologies such as WiFi, WiMax, etc. The network 665 may also be a packet switching network, such as a local area network, a wide area network, a metropolitan area network, the Internet, or other similar types of network environments. The network 665 may be a fixed wireless network, a wireless local area network W (LAN), a wireless wide area network (WWAN), a personal area network (PAN), a virtual private network (VPN), an intranet, or other suitable network systems, and include equipment for receiving and sending signals.

[0056] If the computer 601 is a PC, workstation, smart device, etc., the software in the memory 610 may further include a basic input output system (BIOS) 622. The BIOS is a set of basic software routines that initialize and test the hardware at startup, start the OS 611, and support the transfer of data between hardware devices. The BIOS is stored in the ROM so that the BIOS can be executed when the computer 601 is activated.

[0057] When the computer 601 is running, the processor 605 is configured to execute software 612 stored in the memory 610, transfer data to and from the memory 610, and generally control the operation of the computer 601 according to the software. The methods and OS 611 described herein are read by the processor 605 in whole or in part (but typically the latter), possibly cached within the processor 605, and then executed.

[0058] When the systems and methods described herein are implemented in software 612, such as Figure 5 As shown, the method can be stored on any computer-readable medium, such as storage device 620, for use by or in conjunction with any computer-related system or method. Storage device 620 can include disk storage devices, such as HDD storage devices.

[0059] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0060] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, any suitable combination of electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or the aforementioned storage devices. A non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, such as a punch card or a convex structure in a groove recorded with instructions above, and any suitable combination of the above-mentioned devices. Computer readable storage medium as used herein should not be interpreted as a temporary signal itself, such as radio waves or other free propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (e.g., light pulses by optical fiber cables), or electrical signals transmitted by wires.

[0061] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0062] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and process programming languages ​​(such as "C" programming language or similar programming languages). Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or executed completely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In certain embodiments, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit, so as to perform various aspects of the present invention.

[0063] Various aspects of the present invention are described herein with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to embodiments of the present invention. It will be understood that each frame of the flow chart and / or block diagram and the combination of frames in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0064] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0065] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps will be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0066] Flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in the flow chart or block diagram can represent the module, segment or part of instruction, which includes one or more executable instructions for realizing the specified logical function.In some alternative embodiments, the function noted in the frame may not occur in the order noted in the figure.For example, two frames shown continuously can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the function involved.It will also be noted that the combination of the frames in each frame of the block diagram and / or flow chart illustration and the block diagram and / or flow chart illustration can be realized by a dedicated hardware-based system that performs a specified function or action or performs a combination of special hardware and computer instructions.

[0067] Descriptions of various embodiments have been presented herein for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements existing in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method comprising: Extracting structured information from an unstructured document for unstructured document analysis, wherein extracting structured information for unstructured document analysis comprises: Obtaining a business glossary, the business glossary comprising a list of business terms with definitions; Associating business terms in the business term list with attributes contained in the structured information, the structured information being in the form of a database, and the step includes: identifying tables and columns in the database corresponding to business terms of the business glossary; and For a term in the business glossary and a table identified based on the term in the business glossary, one or more associations between the term in the business glossary and one or more attributes of the identified table; receiving a specification of at least one business term of interest for identification in an unstructured document, wherein the at least one business term of interest comprises one or more business terms in the business glossary, and wherein the at least one business term of interest is associated with a set of at least one attribute of the structured information; generating an analysis module based on the identified tables and columns, the analysis module enabling automatic extraction of structured information corresponding to the business term of interest from the unstructured document based on the designation of the business term of interest by identifying or recognizing attribute values ​​of the attributes of the tables and columns; and using the analysis module to automatically extract structured information from the unstructured document; updating the analysis module based on one or more changes in the database and the business glossary, and The analysis module is continuously updated for extracting structured information from the unstructured document and / or from another unstructured document.

2. The computer-implemented method of claim 1 , wherein: Identifying the tables and columns includes: For each of the plurality of business terms, determining recognition logic based on a format and content of the corresponding business term; and The identification logic is run on the database to identify the tables and columns.

3. The computer-implemented method of claim 1 , wherein: Generating the analysis module comprises: A dictionary of the plurality of business terms is constructed using the attribute values ​​of the identified tables and columns, wherein using the analysis module to extract the structured information includes comparing content of the unstructured document to the dictionary.

4. The computer-implemented method of claim 1 , wherein: Generating the analysis module comprises: Logic is constructed based on the content and format of the attribute values ​​of the identified tables and columns to enable the logic to identify values ​​similar to the attribute values.

5. The computer-implemented method of claim 1 , wherein: If the number of changes is above a threshold, the update is performed.

6. The computer-implemented method of claim 1 , wherein: Extracting the structured information includes: identifying values ​​of the attributes in the unstructured document that correspond to attribute values ​​of the identified tables and columns; and Based on the identified entities of the record, records associated with each entity are formed from the attribute values.

7. The computer-implemented method of claim 6, further comprising: The computer-implemented method is repeated for another unstructured document, wherein the identifying of the tables and columns is performed in the database and in the formed records.

8. The computer-implemented method of claim 1 , wherein: The analysis module is a plug-in.

9. The computer-implemented method of claim 1 , wherein: The database is a master data management MDM database.

10. A computer program product comprising: One or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the program instructions comprising: Program instructions for extracting structured information from an unstructured document for use in unstructured document analysis, wherein the program instructions for extracting structured information for use in unstructured document analysis include: program instructions to obtain a business glossary, the business glossary comprising a list of business terms having definitions; Program instructions for associating business terms in the business glossary with attributes contained in the structured information, the structured information being in the form of a database, and the program instructions comprising: program instructions for identifying tables and columns in the database corresponding to business terms of the business glossary; and program instructions for, for a term in the business glossary and a table identified based on the term in the business glossary, establishing one or more associations between the term in the business glossary and one or more attributes of the identified table; receiving program instructions for specifying at least one business term of interest identified in an unstructured document, wherein the at least one business term of interest comprises one or more business terms in the business glossary, and wherein the at least one business term of interest is associated with a set of at least one attribute of the structured information; generating program instructions for an analysis module based on the identified tables and columns, the analysis module enabling automatic extraction of structured information corresponding to the business term of interest from the unstructured document based on the specification of the business term of interest by identifying or recognizing attribute values ​​of the attributes of the tables and columns; and program instructions for using the analysis module to automatically extract structured information from the unstructured document; program instructions for updating the analysis module based on one or more changes in the database and the business glossary; and The analysis module is continuously updated with program instructions for extracting structured information from the unstructured document and / or from another unstructured document.

11. The computer program product according to claim 10, wherein: The program instructions for identifying the tables and columns include: For each of the plurality of business terms, determining program instructions for recognition logic based on the format and content of the corresponding business term; and Program instructions execute the identification logic on the database to identify the tables and columns.

12. The computer program product of claim 10, wherein: The program instructions for generating the analysis module include: Program instructions for building a dictionary of the plurality of business terms using the attribute values ​​of the identified tables and columns, wherein using the analysis module to extract the structured information includes program instructions for comparing content of the unstructured document to the dictionary.

13. The computer program product of claim 10, wherein: The program instructions for generating the analysis module include: Program instructions are provided to construct logic based on the content and format of the attribute values ​​of the identified tables and columns to enable the logic to identify values ​​similar to the attribute values.

14. A computer system comprising: one or more computer processors; one or more computer-readable storage media; as well as Program instructions stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising: Program instructions for extracting structured information from an unstructured document for use in unstructured document analysis, wherein the program instructions for extracting structured information for use in unstructured document analysis include: program instructions to obtain a business glossary, the business glossary comprising a list of business terms having definitions; Program instructions for associating business terms in the business glossary with attributes contained in the structured information, the structured information being in the form of a database, and the program instructions comprising: program instructions for identifying tables and columns in the database corresponding to business terms of the business glossary; and program instructions for, for a term in the business glossary and a table identified based on the term in the business glossary, establishing one or more associations between the term in the business glossary and one or more attributes of the identified table; receiving program instructions for specifying at least one business term of interest identified in an unstructured document, wherein the at least one business term of interest comprises one or more business terms in the business glossary, and wherein the at least one business term of interest is associated with a set of at least one attribute of the structured information; generating program instructions for an analysis module based on the identified tables and columns, the analysis module enabling automatic extraction of structured information corresponding to the business term of interest from the unstructured document based on the specification of the business term of interest by identifying or recognizing attribute values ​​of the attributes of the tables and columns; and program instructions for using the analysis module to automatically extract structured information from the unstructured document; program instructions for updating the analysis module based on one or more changes in the database and / or the business glossary; and The analysis module is continuously updated with program instructions for extracting structured information from the unstructured document and / or from another unstructured document.

15. The computer system of claim 14, wherein: The program instructions for identifying the tables and columns include: For each of the plurality of business terms, determining program instructions for recognition logic based on the format and content of the corresponding business term; and Program instructions execute the identification logic on the database to identify the tables and columns.

16. The computer system of claim 14, wherein: The program instructions for generating the analysis module include: Program instructions for building a dictionary of the plurality of business terms using the attribute values ​​of the identified tables and columns, wherein using the analysis module to extract the structured information includes program instructions for comparing content of the unstructured document to the dictionary.

17. The computer system of claim 14, wherein: The program instructions for generating the analysis module include: Program instructions are provided to construct logic based on the content and format of the attribute values ​​of the identified tables and columns to enable the logic to identify values ​​similar to the attribute values.

Images