Analyzing deduplicated data chunks associated with unstructured documents

Abstract

This paper describes techniques related to unstructured document processing. One associated computer-implemented method includes: identifying multiple deduplicated data blocks associated with a collection of unstructured documents. The method further includes: sorting the multiple deduplicated data blocks in descending order based on at least one block frequency metric; selecting the highest-ranked unprocessed deduplicated data block; applying text analysis to the selected deduplicated data block; and applying at least one result of the text analysis to any document in the unstructured document collection that includes the selected deduplicated data block. The method terminates in response to the satisfaction of at least one stopping condition.

Description

Technical Field

[0001] The various embodiments described herein generally relate to unstructured document processing. More specifically, the various embodiments describe techniques for processing deduplicated blocks of data associated with collections of unstructured documents in a managed service domain of a cloud computing environment. Summary of the Invention

[0002] The various embodiments described herein provide techniques for unstructured document processing. One associated computer-implemented method includes: identifying a plurality of deduplicated data blocks associated with a collection of unstructured documents; sorting the plurality of deduplicated data blocks in descending order based on at least one block frequency metric; selecting the highest-ranked unprocessed deduplicated data block; applying text analysis to the selected deduplicated data block; and applying at least one result of the text analysis to any document in the collection of unstructured documents that includes the selected deduplicated data block. The method terminates in response to the satisfaction of at least one stopping condition.

[0003] One or more additional embodiments relate to a computer program product including a computer-readable storage medium embodying unstructured document processing program instructions. According to one or more of these additional embodiments, the unstructured document processing program instructions are executable by a computing device to cause the computing device to perform one or more steps associated with the computer-implemented method described above and / or implement one or more embodiments associated with the computer-implemented method described above. One or more further embodiments relate to a system having at least one processor and a memory storing an application that, when executed on the at least one processor, performs unstructured document processing operations. The unstructured document processing operations include one or more steps associated with the computer-implemented method described above and / or implement one or more embodiments associated with the computer-implemented method described above. Attached Figure Description

[0004] To obtain and understand the above aspects in detail, the embodiments briefly outlined above can be described in more detail with reference to the accompanying drawings.

[0005] However, it should be noted that the accompanying drawings only illustrate typical embodiments of the invention and should not be considered as limiting the scope of the invention, as the invention can allow for other equally effective embodiments.

[0006] Figure 1 A cloud computing environment according to one or more embodiments is described.

[0007] Figure 2An abstract model layer provided by a cloud computing environment according to one or more embodiments is described.

[0008] Figure 3 A management service domain in a cloud computing environment according to one or more embodiments is described.

[0009] Figure 4 A method for processing a collection of unstructured documents in a management services domain is illustrated according to one or more embodiments.

[0010] Figure 5 The determination of whether a condition is satisfied according to one or more embodiments is illustrated. Figure 4 The unstructured document processing method shown herein is associated with at least one stopping condition.

[0011] Figure 6 A method for applying text analysis to deduplicated blocks of data selected in the context of unstructured document processing, according to one or more embodiments, is illustrated.

[0012] Figure 7 A method for determining the data sensitivity value of a deduplicated data block selected in the context of unstructured document processing, according to one or more embodiments, is illustrated.

[0013] Figure 8 A method for configuring a text analysis learning model according to one or more embodiments is illustrated. Detailed Implementation

[0014] The various embodiments described herein relate to techniques for processing unstructured documents within the management service domain of a cloud computing environment. In the context of these embodiments, a cloud computing environment is a virtualized environment in which one or more computing capabilities can be used as services. The data processing system of the cloud computing environment associated with these embodiments may optionally utilize the artificial intelligence capabilities of a machine learning knowledge model (specifically, a text analysis learning model) and information from at least one knowledge base associated with such a model.

[0015] Various embodiments may offer advantages over conventional techniques. These embodiments improve computer technology by enabling block-based text analysis instead of document-based text analysis. Specifically, various embodiments utilize deduplicated data blocks to accelerate unstructured document processing by initiating text analysis based on block selection rather than document selection. By applying text analysis to corresponding deduplicated data blocks and applying at least one text analysis result to any unstructured document including the corresponding deduplicated data blocks, various embodiments enable the application of text analysis results to multiple unstructured documents including the corresponding deduplicated data blocks in a single processing iteration, rather than requiring separate processing iterations through multiple unstructured documents, thereby accelerating document processing. Block-based processing according to various embodiments acts as a multiplier in data analysis because it allows for the analysis of larger volumes of data relatively more efficiently over a given time period. Some embodiments in the various examples may not include all of these advantages, and such advantages are not necessarily required in all embodiments.

[0016] In the following text, reference is made to various embodiments of the invention. However, it should be understood that the invention is not limited to the specifically described embodiments. Rather, any combination of the following features and elements (whether or not related to different embodiments) is considered to realize and practice the invention. Furthermore, while embodiments may achieve advantages over other possible solutions and / or over the prior art, it is not limiting whether a given embodiment achieves a particular advantage. Therefore, the following aspects, features, embodiments, and advantages are merely illustrative and should not be considered elements or limitations of the appended claims unless expressly stated in the claims. Similarly, references to “the invention” should not be construed as a generalization of any inventive subject matter disclosed herein and should not be considered elements or limitations of the appended claims unless expressly stated in one or more claims.

[0017] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0018] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0019] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0020] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including local area network (LAN) or wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing the status information of the computer-readable program instructions.

[0021] The aspects of the present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of manufacture comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0022] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0023] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be implemented as a single step, executed simultaneously, substantially simultaneously, with partial or complete time overlap, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0024] Specific embodiments are described herein, relating to techniques related to unstructured document processing in a management services domain. However, it should be understood that the techniques described herein are applicable to a variety of purposes in addition to those specifically described herein. Therefore, references to specific embodiments are intended to be illustrative rather than restrictive.

[0025] The various embodiments described herein can be provided to end users via cloud computing infrastructure. It should be understood that while this disclosure includes a detailed description of cloud computing, implementations of the teachings cited herein are not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0026] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, storage, applications, VMs, and services) that can be rapidly provisioned and released with minimal management costs or interaction with service providers. Therefore, cloud computing allows users to access virtual computing resources (e.g., storage, data, applications, and even complete virtualized computing systems) in the cloud, regardless of the underlying physical systems (or the location of those systems) used to provide those computing resources. This cloud model can include at least five features, at least three service models, and at least four deployment models.

[0027] The features are as follows:

[0028] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power (such as server time and network storage) on demand without human interaction with the service provider.

[0029] Wide network access: Capabilities are available on the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0030] Resource pooling: A provider's computing resources are grouped into resource pools to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Typically, consumers cannot control or know the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center), thus exhibiting location independence.

[0031] Rapid flexibility: Capabilities can be rapidly and flexibly (in some cases automatically) provided to expand outward quickly and be rapidly released to shrink back down. For consumers, the available capacity often appears unlimited and can be purchased at any time and in any quantity.

[0032] Measurable services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0033] The service model is as follows:

[0034] Software as a Service (SaaS): The capability offered to consumers is the ability to use applications running on a provider's cloud infrastructure. These applications can be accessed from various client devices via thin client interfaces such as web browsers (e.g., web-based email). Aside from limited user-specific application configuration settings, consumers neither manage nor control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities.

[0035] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications on cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers neither manage nor control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the applications they deploy and may also have control over the configuration of the application hosting environment.

[0036] Infrastructure as a Service (IaaS): This provides consumers with the capability to deploy and run any software, including operating systems and applications, on the cloud, providing them with processing, storage, networking, and other basic computing resources. Consumers neither manage nor control the underlying cloud infrastructure, but they have control over the operating system, storage, and deployed applications, and may have limited control over chosen network components (e.g., host firewalls).

[0037] The deployment model is as follows:

[0038] Private cloud: A cloud infrastructure that runs exclusively for a single organization. It can be managed by that organization or a third party, and can exist inside or outside the organization.

[0039] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist inside or outside the organization.

[0040] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0041] Hybrid cloud: A cloud infrastructure consisting of two or more clouds (private, community, or public) that remain distinct entities but are bound together by standardized or proprietary technologies that enable data and applications to be ported together (e.g., cloud bursts for load balancing between clouds).

[0042] Cloud computing environments are service-oriented, characterized by statelessness, loose coupling, modularity, and semantic interoperability. The core of computing is the infrastructure comprising a network of interconnected nodes.

[0043] Figure 1 A cloud computing environment 50 according to one or more embodiments is illustrated. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers can communicate. Examples of such local computing devices include, but are not limited to, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. Nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds as described above, or combinations thereof. This allows the cloud computing environment 50 to provide Infrastructure as a Service, Platform as a Service, and / or Software as a Service, without requiring cloud consumers to maintain resources for them on their local computing devices. It should be understood that... Figure 1The types of computing devices 54A-N shown are merely illustrative, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connectivity (e.g., using a web browser).

[0044] Figure 2 A set of functional abstraction layers provided by a cloud computing environment 50 according to one or more embodiments is illustrated. It should be understood in advance that... Figure 2 The components, layers, and functions shown are merely illustrative, and embodiments of the invention are not limited thereto. As described, various layers and corresponding functions are provided. Specifically, hardware and software layer 60 includes hardware and software components. Examples of hardware components may include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68. Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: a virtual server 71; virtual storage 72; a virtual network 73, including a virtual private network; virtual applications and operating systems 74; and virtual clients 75.

[0045] In one example, management layer 80 can provide the following functionalities: Resource Provisioning 81: Provides dynamic acquisition of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82: Provides cost tracking for the use of resources in the cloud computing environment and provides bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security Functions: Provides authentication for cloud consumers and tasks and protection for data and other resources. User Portal 83: Provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84: Provides cloud resource allocation and management to meet required service levels. Service Level Agreement (SLA) Planning and Fulfillment 85: Provides pre-scheduling and procurement of cloud resources according to the SLA for its projected future needs.

[0046] Workload layer 90 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided in this layer include, but are not limited to: map creation and navigation 91; software development and lifecycle management 92; virtual classroom instruction provision 93; data analysis and processing 94; transaction processing 95; and unstructured document processing 96. Unstructured document processing 96 enables the processing of deduplicated blocks of data associated with a collection of unstructured documents, according to the various embodiments described herein.

[0047] Figure 3The diagram illustrates a management service domain 300 within a cloud computing environment 50. Functions related to unstructured document processing 96 and other workloads / functions can be performed within the management service domain 300. The management service domain 300 includes a data processing system 310, a document storage system 320, one or more external database systems 330, and multiple application server clusters 3401 to 340. n As shown in the figure, the data processing system 310 includes a data processing application 350. The data processing application 350 includes a text analysis learning model 355, which incorporates machine learning knowledge model capabilities. The data processing application 350 represents a single application or multiple applications. The text analysis learning model 355 is configured to facilitate unstructured document processing according to the various embodiments described herein. In alternative embodiments, one or more aspects of the data processing system 310 are integrated into a hardware-based local server infrastructure. According to this alternative embodiment, such one or more aspects of the data processing system 310 are more generally connected via at least one aspect of a network-based connectivity and management service domain 300 and / or a cloud computing environment 50. In a further embodiment, the data processing system 310 is configured to interact with a document storage system 320, one or more external database systems 330, and multiple application server clusters 3401 to 340. n Communication. Additionally, application server clusters 3401 to 340... n The individual servers within can optionally be configured to communicate with each other and / or with server clusters in other domains.

[0048] As shown in the figure, document storage system 320 includes management server 360, document storage 365, and document management system 370. Management server 360 is configured to communicate with other aspects of management service domain 300 (including data processing system 310). In one embodiment, document storage 365 includes at least one database, optionally at least one relational database. Additionally or alternatively, document storage 365 includes at least one file system, optionally at least one file transfer protocol (FTP) system. In another embodiment, document storage 365 stores data associated with at least one knowledge base of text analysis learning model 355. In a further embodiment, document management system 370 is or includes a database management system (DBMS), optionally a relational database management system (RDBMS). In a further embodiment, document management system 370 manages one or more external database systems 330 or otherwise interacts with one or more external database systems 330. In a further embodiment, document management system 370 includes one or more ontology trees or other ontology structures. In a further embodiment, document management system 370 coordinates and manages at least one knowledge base of text analysis learning model 355. The document management system 370 is configured to manage the storage of physical and / or logical data blocks. In an alternative embodiment, some or all aspects of the document management system 370 are integrated into a management server 360. In a further alternative embodiment, one or more aspects of the document storage system 320 are integrated into a hardware-based local server infrastructure. According to such a further alternative embodiment, such one or more aspects of the document storage system 320 are more generally connected to one or more aspects of the management service domain 300 and / or cloud computing environment 50 via at least one network-based connection.

[0049] One or more external database systems 330 may optionally include at least one database / relational database or at least one DBMS / RDBMS configured to interface with the document management system 370. In a further embodiment, the document management system 370 and / or at least one DBMS / RDBMS included in one or more external database systems 330 store multiple application server clusters 3401 to 340. n Relationships with at least one knowledge base. Application server cluster 3401 to 340 n It is configured to host and / or store aspects of various applications, and is also configured to provide management services to one or more client systems and / or data systems (including data processing system 310 or document storage system 320).

[0050] Figure 4An unstructured document processing method 400 is illustrated. In one embodiment, one or more steps associated with method 400 are performed in an environment where computing power is provided as a service (e.g., cloud computing environment 50). According to this embodiment, one or more steps associated with method 400 are performed in a management service domain (e.g., management service domain 300) within that environment. The environment may optionally be a hybrid cloud environment. In a further embodiment, one or more steps associated with method 400 are performed in one or more other environments (such as a client-server network environment or a peer-to-peer network environment). A data processing system in the management service domain (e.g., data processing system 310) may facilitate processing according to method 400 and other methods further described herein. More specifically, a data processing application in the data processing system (e.g., data processing application 350) may perform or otherwise facilitate one or more steps of method 400 and other methods described herein. Unstructured document processing techniques facilitated or otherwise performed by data processing systems in the management services domain can be associated with unstructured data processing within workloads in functional abstraction layers provided by the environment (e.g., unstructured document processing 96 within workload layer 90 of cloud computing infrastructure 50).

[0051] Method 400 begins at step 405, wherein the data processing application identifies multiple deduplicated data blocks associated with a collection of unstructured documents. The collection of unstructured documents may optionally be stored in or associated with a document storage system (e.g., document storage system 320) within or associated with a management service domain and / or accessible via such a system. The collection of unstructured documents may be referred to as a data corpus. In embodiments, the multiple deduplicated data blocks include physical data blocks corresponding to portions of physical data storage and / or storage from, for example, a document storage system. Additionally or alternatively, the multiple deduplicated data blocks include logical data blocks, i.e., virtualized data blocks, managed by the data processing application and / or at least one content management application associated with the document storage system. These logical data blocks are organized and / or managed regardless of the physical data storage or physical storage layout, as they are virtualized via application software. Storing multiple deduplicated data blocks avoids the separate and unnecessary storage of duplicate data blocks, as further described herein, enabling more efficient data analysis. Each of the multiple deduplicated data blocks is configurable in size, optionally measured in bytes.

[0052] The plurality of deduplicated data blocks identified in step 405 may optionally be stored in and / or accessible via a document storage system. In an embodiment, the data processing application identifies one or more deduplicated data blocks from the plurality of deduplicated data blocks according to step 405 through communication with the document storage system. The document storage system includes at least one management server (e.g., management server 360) configured to interface with the data processing application and / or other aspects of the data processing system. The document storage system may optionally include one or more databases (e.g., incorporated into and / or accessible via document storage 365). Additionally or alternatively, the document storage system includes one or more file server systems, such as FTP systems (e.g., incorporated into and / or accessible via document storage 365). The document storage system also includes at least one document management system (e.g., document management system 370) configured to store, access, or otherwise manage documents (including collections of unstructured documents) and other data in corresponding storage locations defined by a system-specific implementation. Such storage locations include physical blocks and / or logical blocks. In this embodiment, some or all aspects of the document storage system are cloud-based.

[0053] In step 410, the data processing application sorts a plurality of deduplicated data blocks in descending order based on at least one block frequency metric. In an embodiment, the data processing application sorts the plurality of deduplicated data blocks by creating a reference list comprising the plurality of deduplicated data blocks, the plurality of deduplicated data blocks in the reference list being sorted in descending order based on at least one block frequency metric. The reference list may optionally be stored as a data structure, such as a linked list or a sorted array. In a related embodiment, the reference list indexes the deduplicated data blocks by both document and document position, such that the reference list for each deduplicated data block includes both the record of each document containing the deduplicated data block in the unstructured document set and the record of the deduplicated data block at one or more corresponding positions within each document. In an alternative related embodiment, the reference list indexes the deduplicated data blocks only by document, in which case the reference list for each deduplicated data block includes only the record of each document containing the deduplicated data block in the unstructured document set, excluding the record at one or more corresponding block positions within each document.

[0054] In one embodiment, at least one block frequency metric includes unique document usage frequency. According to this embodiment, a data processing application sorts multiple deduplicated data blocks in descending order of unique document usage frequency. In the context of various embodiments, the unique document usage frequency of a deduplicated data block is defined as the number of unique documents to which the deduplicated data block resides. The unique document frequency value is equal to the number of documents that include at least one instance of the deduplicated data block. The unique document usage frequency measures the frequency of block sharing among documents. The unique document usage frequency of a deduplicated data block is proportional to the sharing frequency of the deduplicated data block within the unstructured document set. Optionally, the unique document usage frequency of a deduplicated data block is expressed as an integer value corresponding to the number of documents to which the block resides in the unstructured document set. Alternatively, the unique document usage frequency of a deduplicated data block is expressed as a percentage value corresponding to the percentage of documents to which the block resides in the unstructured document set. In an additional embodiment, at least one block frequency metric includes unique block occurrence frequency. According to this additional embodiment, a data processing application sorts multiple deduplicated data blocks in descending order of unique block occurrence frequency. In the context of various embodiments, the unique occurrence frequency of a deduplicated data block is defined as the number of times the deduplicated data block appears uniquely within the unstructured document set. In data processing scenarios where a deduplicated data block is used at most once within any document in the unstructured document set, this unique occurrence frequency of the deduplicated data block is equal to the unique document usage frequency of this deduplicated data block. Optionally, the unique occurrence frequency of a deduplicated data block is expressed as an integer value corresponding to the number of times the block appears within the unstructured document set.

[0055] In one embodiment, the data processing application sorts multiple deduplicated data blocks in descending order based on only one block frequency metric in step 410. According to this embodiment, the data processing application may optionally sort the multiple deduplicated data blocks in descending order based solely on unique document usage frequency values ​​or solely on unique block occurrence frequency values. In an additional embodiment, the data processing application sorts the multiple deduplicated data blocks in descending order based on a combination of multiple block frequency metrics. According to this additional embodiment, the data processing application may optionally sort the multiple deduplicated data blocks in descending order in step 410 by aggregating (e.g., by summing or averaging) the corresponding unique document usage frequency value and the corresponding unique block occurrence frequency value for each of the multiple deduplicated data blocks. Specifically, for each of the multiple deduplicated data blocks, the data processing application may optionally sum or average the corresponding unique document usage frequency value (which reflects the number of unique documents to which the block resides, e.g., an integer value) and the corresponding unique block occurrence frequency value (which reflects the number of times the block occurs within the unstructured document set, e.g., an integer value).

[0056] In an embodiment, the data processing application obtains one or more block frequency metrics from at least one block frequency metric of the corresponding deduplicated data block by querying the index file, such as unique document usage frequency value and / or unique block occurrence frequency value. The index file optionally includes one or more other metrics associated with one or more deduplicated data blocks among a plurality of deduplicated data blocks. In an additional embodiment, the index file includes an inverted index data structure storing a mapping from each deduplicated data block to one or more documents within the unstructured document set. The inverted index mapping optionally indicates the existence of a deduplicated data block within one or more documents within the unstructured document set. Furthermore, the inverted index mapping optionally includes information about the location of the deduplicated data block within one or more documents within the unstructured document set. The location information includes the absolute data block location within the document and / or the data block location within the document relative to other data blocks. Optionally, the index file is an implementation of the previously described reference list created in the context of sorting according to step 410.

[0057] In step 415, the data processing application selects the highest-ranked unprocessed deduplicated data block from a plurality of deduplicated data blocks. According to step 415, the data processing application selects the still-to-be-selected deduplicated data block that is ranked highest based on at least one block frequency metric. The unprocessed deduplicated data blocks in the context of step 415 are blocks that the data processing application has not yet selected for text analysis. According to an embodiment where the data processing application creates a descendingly sorted list of references based on at least one block frequency metric, the data processing application selects the unprocessed deduplicated data block by selecting the unprocessed deduplicated data block at the highest position in the reference list. By selecting the unprocessed deduplicated data block with the highest ranking value based on at least one block frequency metric, the data processing application prioritizes the analysis of unstructured content in the unstructured document collection with the highest block frequency of the deduplicated data block, where processing continues in descending order based on at least one block frequency metric. By prioritizing deduplicated data blocks based on at least one block frequency metric, insights gained from deduplicated data block analysis can be applied relatively quickly to a relatively large number of documents (or document portions). Based on steps 410 and 415, the data processing application selects deduplicated data blocks in order from highest to lowest ranking value based on at least one block frequency metric.

[0058] In step 420, the data processing application applies text analysis to the selected deduplicated data block. In an embodiment, the data processing application applies text analysis by facilitating the application of at least one Natural Language Processing (NLP) technique to the selected deduplicated data block. The at least one NLP technique may optionally be combined with Natural Language Understanding (NLU). The data processing application may optionally apply NLP techniques and / or optionally facilitate application programming interface (API) calls to applications with NLP capabilities (e.g., at least one cloud-based NLP application). The data processing application may optionally apply NLP for the purpose of contextual analysis and / or logical relational analysis. In the context of various embodiments, contextual analysis of the data block includes analyzing one or more text elements of the data block, taking into account one or more other text elements of the data block. In the context of a retrospective embodiment, logical relational analysis of the data block includes analyzing at least one measurable correlation between or among the text elements of the data block. In a related embodiment, the data processing application facilitates the application of at least one NLP technique by applying a recurrent neural network (RNN) model to aspects of the selected deduplicated data block to establish machine learning (deep learning)-based connections (e.g., contextual connections and / or logical relational connections) between data points within the deduplicated data block. In a further embodiment, when identifying audio (such as spoken words) within a selected deduplicated data block, the data processing application may optionally apply at least one automatic speech recognition (ASR) technique (e.g., speech-to-text) to the selected deduplicated data block to derive text-based aspects from the audio, and subsequently apply NLP to the derived text-based aspects. In a further embodiment, when identifying visual images (such as still images and / or videos of user activity or activity of associated contacts) within a selected deduplicated data block, the data processing application may optionally apply video recognition (e.g., video-to-text) to the selected deduplicated data block to derive text-based aspects from the visual images, and subsequently apply NLP to the derived text-based aspects.

[0059] In one embodiment, the data processing application applies text analysis to identify data attributes within a selected deduplicated data block. For example, the data processing application may optionally identify all data associated with a specific entity within a selected pair of deduplicated data blocks. According to this embodiment, the data processing application identifies data points associated with a specific entity, including data access characteristics such as data access instances and / or data access patterns. In the context of various embodiments, an entity may be an individual, a group of individuals, or an organization. In a further embodiment, the data processing application applies text analysis to identify data security aspects (optionally, including data security risk factors) within the selected deduplicated data block. According to this further embodiment, the data processing application identifies sensitive data. Such sensitive data may optionally include confidential data and / or data of relatively high interest to at least one entity associated with one or more documents in a collection of unstructured documents. Such sensitive data may optionally include personal information associated with an individual, personal information associated with one or more individuals in a group of individuals, and / or organizational information associated with a company, institution, collection, or other group.

[0060] In one embodiment, the data processing application applies contextual analysis to a selected deduplicated data block. In a further embodiment, the data processing application applies logical relationship analysis to the selected deduplicated data block. This embodiment is applicable to deduplicated data blocks comprising multiple bytes. A deduplicated data block of multiple bytes size includes sufficient data to allow analysis of the context between block n-grams or other block portions (e.g., words or other data aspects) and / or the logical relationships between block n-grams or other block portions. In the context of various embodiments, an n-gram is defined as a continuous sequence of "n" items in a document. The data processing application applies contextual analysis and / or logical relationship analysis to facilitate the characterization of data attributes and / or the identification of data security aspects such as sensitive information. The data processing application applies contextual analysis and / or logical relationship analysis to identify the contextual and / or logical relationships between entities within the selected deduplicated data block and at least one associated n-gram. Analysis of an entity alone with respect to an entity associated with the selected deduplicated data block may not provide context regarding potentially sensitive information associated with that entity. However, the application of contextual analysis and / or logical relational analysis between such an entity and at least one associated n-gram can indicate the sensitivity of information associated with such an entity. For example, while identifying a date within a selected deduplicated data block alone may not lead to significant analytical results, analysis of such a date, along with relevant contextual and / or logical relational information associated with the block, can reveal that such a date is sensitive to the entity or has other importance, such as an individual's birthday. In another example, while identifying a name within a selected deduplicated data block alone may not lead to significant analytical results, analysis of such a name, along with relevant contextual and / or logical relational information associated with the block, can reveal that such a name is sensitive in the context of one or more entities associated with the block. As further described herein, the data processing application may optionally apply contextual analysis and / or logical relational analysis to configure the learning model based on textual analysis for the purpose of assessing data block sensitivity, including determining whether the selected deduplicated data block is classified as sensitive. According to step 420, the data processing application initiates textual analysis based on the selection of deduplicated data blocks rather than document selection. The data processing application applies text analysis to the selected deduplicated data block once, according to step 420, rather than every time the selected deduplicated data block appears in a document within the unstructured document collection. About Figure 6 A method is described for applying text analysis to the selected deduplicated data block according to step 420.

[0061] In step 425, the data processing application applies at least one result of the text analysis to any document in the unstructured document set that includes the selected deduplicated data block. In an embodiment, the data processing application identifies any document in the unstructured document set that includes the selected deduplicated data block by, for example, retrieving any document identification data associated with the selected deduplicated data block from the previously discussed citation list (including document identification parameters that associate documents in the unstructured document set with the block). According to an embodiment in which the data processing application applies text analysis to the selected deduplicated data block to determine data attributes within the block, applying at least one result of the text analysis may optionally include: labeling or otherwise marking aspects of any document or portion thereof that includes the block based on the determined data attributes. In an embodiment, the data processing application first labels the document data based on the determined data attributes, and then characterizes the labeled data by applying at least one supervised machine learning classification technique (e.g., by applying at least one classification algorithm) or by applying at least one unsupervised machine learning clustering technique (e.g., by applying at least one clustering algorithm). For example, data processing applications may optionally tag documents based on identifiers associated with an individual's birthday, and then use these tags to specify document types and / or classify or cluster relevant document aspects.

[0062] According to an embodiment in which a data processing application applies text analysis to a selected deduplicated data block to identify data security aspects, at least one result of applying the text analysis in step 425 may optionally include addressing any such data security aspect within any document or portion thereof that includes the block. In a related embodiment, addressing any such data security aspect may optionally include removing or isolating any data associated with one or more data security aspects within any document that includes the selected deduplicated data block or at any document location that includes the selected deduplicated data block. In a further related embodiment, addressing any such data security aspect may optionally include tagging or otherwise marking any document or portion thereof that includes the selected deduplicated data block to identify any data security risk factors. As further described herein, in response to classifying the selected deduplicated data block as sensitive, addressing any such data security aspect may optionally include classifying any document or portion thereof that includes the selected deduplicated data block as sensitive. By applying at least one result according to step 425 to any document including the selected deduplicated data block, various embodiments enable analysis to be performed in a single processing iteration and the analysis results applied to multiple documents including the selected deduplicated data block within a collection of unstructured documents, rather than requiring separate processing iterations through each of the multiple documents including the selected deduplicated data block. By reducing processing iterations, the block-based text analysis of various embodiments accelerates unstructured document analysis. Furthermore, efficiency is improved due to the lower resource consumption resulting from this streamlined block-based text analysis.

[0063] In an alternative embodiment, the data processing application applies the steps of method 400 to deduplicated data blocks within corresponding portions of a single unstructured document. According to this alternative embodiment, the data processing application applies text analysis to the selected deduplicated data blocks in step 420, and applies at least one result of the text analysis to any document portion of the single unstructured document that includes the selected deduplicated data blocks in step 425. According to this alternative embodiment, by applying at least one result to any portion of a single unstructured document that includes the selected deduplicated data blocks, various embodiments allow text analysis to be performed in a single processing iteration and the text analysis results to multiple portions of a single unstructured document that include the selected deduplicated data blocks, rather than requiring separate processing iterations through each of the multiple portions that include the selected deduplicated data blocks.

[0064] In step 430, the data processing application determines whether there exists at least one unprocessed deduplicated data block among a plurality of deduplicated data blocks to be selected. According to step 430, the data processing application determines whether there exists at least one deduplicated data block that has not yet been selected for the text analysis application. In response to determining that there is no unprocessed deduplicated data block to be selected (e.g., since all deduplicated data blocks have been selected), the data processing application continues to the end of method 400. In response to determining that there is at least one unprocessed deduplicated data block to be selected, the data processing application continues to step 435.

[0065] In step 435, the data processing application determines whether at least one stopping condition is met. In the context of method 400, a stopping condition is a condition that commands the data processing application to terminate the processing of deduplicated data blocks. In response to determining that at least one stopping condition is met, the data processing application continues to the end of method 400. Therefore, in response to meeting at least one stopping condition, the data processing application terminates method 400. In response to determining that no stopping condition is met, the data processing application returns to step 415 to select the highest-ranked unprocessed deduplicated data block. About Figure 5 A method is described for determining whether at least one stopping condition is met according to step 435.

[0066] Figure 5A method 500 for determining whether at least one stopping condition is met is illustrated. Method 500 provides one or more embodiments of step 435 of method 400. Method 500 begins at step 505, where a data processing application determines whether the unique document usage frequency of the next unprocessed deduplicated data block to be selected is below a predetermined document influence threshold. The data processing application identifies the next unprocessed deduplicated data block to be selected from a plurality of deduplicated data blocks by determining the highest-ranking unprocessed deduplicated data block based on at least one block frequency metric (e.g., by querying a previously described citation list). In response to determining that the unique document usage frequency of the next unprocessed deduplicated data block to be selected is below the predetermined document influence threshold, the data processing application continues to step 530, where the data processing application determines that at least one stopping condition is met, and continues to the end of step 500. As a result of determining that at least one stopping condition is met according to step 530, the data processing application continues to the end of method 400 according to step 435, thereby terminating further unstructured document processing. Therefore, as a result of executing steps 505 and 530 in step 435 if at least one stopping condition is met, the data processing application terminates the unstructured document processing method 400 in response to determining that the frequency of occurrence of the unique block of the next unprocessed deduplicated data block to be selected is lower than a predetermined document influence threshold. In an embodiment, as a result of executing steps 505 and 530, the data processing application terminates method 400 before applying text analysis to the next unprocessed deduplicated data block to be selected and before applying at least one result of text analysis to any document in the unstructured document set that includes the block. Terminating method 400 based on the predetermined document influence threshold can improve method efficiency because the data processing application can avoid analyzing one or more unselected deduplicated data blocks that have a relatively small document influence compared to the previously selected deduplicated data blocks. In response to determining that the frequency of occurrence of the unique document of the next unprocessed deduplicated data block to be selected is not lower than the predetermined document influence threshold, the data processing application continues to step 510. In a further embodiment, the data processing application applies the unique document usage frequency as a stopping condition according to step 505, even if the unique document usage frequency is not among at least one block frequency metric applied for the purpose of sorting multiple deduplicated data blocks in step 410.

[0067] In step 510, the data processing application determines whether the unique block occurrence frequency of the next unprocessed deduplicated data block to be selected is lower than a predetermined block occurrence threshold. In an embodiment, the predetermined block occurrence threshold is equal to a predetermined document influence threshold. In response to determining that the unique block occurrence frequency of the next unprocessed deduplicated data block to be selected is lower than the predetermined block occurrence threshold, the data processing application continues to step 530, in which the data processing application determines that at least one stopping condition is met, and continues to the end of step 500. As a result of determining that at least one stopping condition is met according to step 530, the data processing application continues to the end of method 400 according to step 435, thereby terminating further unstructured document processing. Therefore, as a result of executing steps 510 and 530 in the case of determining whether at least one stopping condition is met in step 435, in response to determining that the unique block occurrence frequency of the next unprocessed deduplicated data block to be selected is lower than the predetermined block occurrence threshold, the data processing application terminates the unstructured document processing method 400. In response to determining that the unique block occurrence frequency of the next unprocessed deduplicated data block to be selected is not lower than the predetermined block occurrence threshold, the data processing application continues to step 515. In a further embodiment, the data processing application applies the unique block occurrence frequency as a stopping condition according to step 510, even if the unique block occurrence frequency is not among at least one block frequency metric applied for the purpose of sorting multiple deduplicated data blocks in step 410.

[0068] In step 515, the data processing application determines whether a predetermined unstructured document evaluation period has expired. In this embodiment, the unstructured document evaluation period is a fixed duration. In response to determining that the predetermined unstructured document evaluation period has expired, the data processing application proceeds to step 530, where it determines that at least one stopping condition is met and continues to the end of step 500. As a result of determining that at least one stopping condition is met according to step 530, the data processing application continues to the end of method 400 according to step 435, thereby terminating further unstructured document processing. Therefore, as a result of executing steps 515 and 530 in the case of determining whether at least one stopping condition is met in step 435, in response to determining that the predetermined unstructured document evaluation period has expired, the data processing application terminates the unstructured document processing method 400. In response to determining that the predetermined unstructured document evaluation period has not yet expired, the data processing application continues to step 520. In an alternative embodiment, upon determining that the predetermined unstructured document evaluation period has expired, the data processing application immediately terminates method 400, and further immediately terminates method 500 (if applicable).

[0069] In step 520, the data processing application determines whether the number of deduplicated data blocks among a plurality of deduplicated data blocks that have undergone text analysis exceeds a predetermined block text analysis threshold. In one embodiment, the number of deduplicated data blocks among a plurality of deduplicated data blocks is the number of blocks compared to the predetermined block text analysis threshold, in which case the predetermined block text analysis threshold is an integer value. In an alternative embodiment, the number of deduplicated data blocks among a plurality of deduplicated data blocks is a percentage of blocks compared to the predetermined block text analysis threshold, in which case the predetermined block text analysis threshold is a percentage value. In response to determining that the number of deduplicated data blocks among a plurality of deduplicated data blocks that have undergone text analysis exceeds the predetermined block text analysis threshold, the data processing application continues to step 530, in which the data processing application determines that at least one stopping condition is met, and continues to the end of step 500. As a result of determining that at least one stopping condition is met according to step 530, according to step 435, the data processing application continues to the end of method 400, thereby terminating further unstructured document processing. Therefore, as a result of executing steps 520 and 530 in step 435 if at least one stopping condition is met, in response to determining that the number of deduplicated data blocks among the plurality of deduplicated data blocks to which text analysis has been applied exceeds a predetermined block text analysis threshold, the data processing application terminates the unstructured document processing method 400. According to step 520, in response to determining according to step 420 that the number of deduplicated data blocks that have been selected and processed exceeds the predetermined block text analysis threshold, the data processing application terminates the unstructured document processing method 400. In response to determining that the number of deduplicated data blocks among the plurality of deduplicated data blocks to which text analysis has been applied does not exceed the predetermined block text analysis threshold, the data processing application continues to step 525.

[0070] In step 525, the data processing application determines whether the number of documents in the unstructured document set that have been applied at least one text analysis result exceeds a predetermined analysis result allocation threshold. In one embodiment, the number of documents in the unstructured document set is the number of documents compared to the predetermined analysis result allocation threshold, in which case the predetermined analysis result allocation threshold is an integer value. In an alternative embodiment, the number of documents in the unstructured document set is a percentage of documents compared to the predetermined analysis result allocation threshold, in which case the predetermined analysis result allocation threshold is a percentage value. In response to determining that the number of documents in the unstructured document set that have been applied at least one text analysis result exceeds the predetermined analysis result allocation threshold, the data processing application continues to step 530, whereby the data processing application determines that at least one stopping condition is met, and continues to the end of step 500. As a result of determining that at least one stopping condition is met according to step 530, the data processing application continues to the end of method 400 according to step 435, thereby terminating further unstructured document processing. Therefore, as a result of executing steps 525 and 530 in step 435 if it is determined whether at least one stopping condition is met, in response to determining that the number of documents in the unstructured document set to which at least one text analysis result has been applied exceeds a predetermined analysis result allocation threshold, the data processing application terminates the unstructured document processing method 400. According to step 525, in response to determining according to step 425 that the number of processed documents exceeds the predetermined analysis result allocation threshold, the data processing application terminates the unstructured document processing method 400. In response to determining that the number of documents in the unstructured document set to which at least one text analysis result has been applied does not exceed the predetermined analysis result allocation threshold, the data processing application continues to step 535, where the data processing application determines that the stopping condition has not been met. As a result of determining according to step 535 that no stopping condition is met, according to step 435, the data processing application returns to step 415 to select the highest-ranked unprocessed deduplicated data block.

[0071] At each step of steps 505 to 525, the data processing application applies a corresponding stop condition. In one or more embodiments, the data processing application applies the corresponding stop conditions of steps 505 to 525 in any order. Optionally, the data processing application determines the application order of multiple stop conditions based at least in part on input obtained from at least one external entity (e.g., a data processing system administrator and / or a data processing system client). Optionally, one or more thresholds among the corresponding thresholds applied at steps 505 to 525 are predetermined by the data processing application and / or predetermined based on input obtained from at least one external entity. In one or more additional embodiments, the data processing application applies only a subset of the stop conditions of steps 505 to 525 to determine whether a stop condition is satisfied. The data processing application may optionally apply only a single stop condition among the stop conditions of steps 505 to 525 to determine whether a stop condition is satisfied. Optionally, the data processing application determines whether to apply all stop conditions or a subset of stop conditions based at least in part on input obtained from at least one external entity.

[0072] Figure 6A method 600 for applying text analytics to a selected deduplicated data block is illustrated. Method 600 provides one or more embodiments of step 420 of method 400. Method 600 begins at step 605, where a data processing application determines a data sensitivity value for the selected deduplicated data block by evaluating data within the selected deduplicated data block, taking into account a text analytics learning model (e.g., text analytics learning model 355). In an embodiment, the data processing application determines the data sensitivity value according to step 605 based at least in part on archived sensitive data associated with the text analytics learning model. According to this embodiment, the data processing application evaluates the data within the selected deduplicated data block by facilitating a comparison between the block data and the archived sensitive data. The data processing application may optionally determine a relatively high data sensitivity value for the selected deduplicated data block in response to determining a relatively high level of similarity between the block data and the archived sensitive data. The archived sensitive data relates to one or more entities associated with one or more documents in a collection of unstructured documents. Optionally, the archived sensitive data includes personal information relating to one or more entities. The data processing application identifies and evaluates any personal information associated with a selected deduplicated data block based on comparisons with personal information included in the archived sensitive data. Additionally or alternatively, the archived sensitive data includes confidential information relating to one or more entities, such as authentication information (e.g., password data). The data processing application identifies and evaluates any confidential information associated with a selected deduplicated data block based on comparisons with confidential information included in the archived sensitive data. As further described herein, the data processing application facilitates the training of a text analytics learning model based on the archived sensitive data. The comparisons discussed in step 605 may optionally include direct text comparisons or comparisons of text patterns, as described below.

[0073] In a related embodiment, the data processing application determines a data sensitivity value in step 605 based at least in part on at least one sensitive data pattern associated with the text analysis learning model. As further described herein, the data processing application derives at least one sensitive data pattern when configuring the text analysis learning model. According to this related embodiment, the data processing application evaluates data within a selected deduplication block by facilitating a comparison between any identified pattern within the block of data and at least one sensitive data pattern. Optionally, in response to determining a relatively high correlation between any identified pattern within the block of data and at least one sensitive data pattern, the data processing application determines a relatively high data sensitivity value for the selected deduplication block.

[0074] In one embodiment, the data sensitivity value determined for the deduplicated data block in step 605 is quantitative. According to this embodiment, the quantitative data sensitivity value may optionally be normalized, and / or optionally on a predefined scale, for example, between 0 and 1, where 0 represents the lowest sensitivity and 1 indicates the highest sensitivity. In an additional embodiment, the data processing application determines the data sensitivity value of the selected deduplicated data block by determining the corresponding data sensitivity value for the corresponding portion of the selected deduplicated data block. Regarding Figure 7 A method is described for determining the data sensitivity value of the selected deduplicated data block according to step 605.

[0075] In step 610, the data processing application determines whether the data sensitivity value of the selected deduplicated data block exceeds a sensitive information threshold. In an embodiment, the sensitive information threshold may optionally be on a predetermined scale for comparison with the data sensitivity value. In an additional embodiment, the data processing application predefines the sensitive information threshold based at least in part on input obtained from at least one external entity (e.g., a data processing system administrator or a data processing system client). In response to determining that the data sensitivity value of the selected deduplicated data block does not exceed the sensitive information threshold, the data processing application continues to the end of method 600. In response to determining that the data sensitivity value of the selected deduplicated data block exceeds the sensitive information threshold, in step 615, the data processing application classifies the selected Dee deduplicated data block as sensitive.

[0076] When a selected deduplicated data block is classified as sensitive, the data processing application may optionally tag or otherwise mark the selected deduplicated data block, for example, in the previously described reference list. In an embodiment, when at least one result of text analysis is applied to any document in the unstructured document set that includes the selected deduplicated data block according to step 425, the data processing application classifies any document, or at least a portion thereof, that includes the selected deduplicated data block classified as sensitive. In a further embodiment, the data processing application implements access control for any document, or at least a portion thereof, that includes the selected deduplicated data block classified as sensitive.

[0077] In summary, applying text analysis to the selected deduplicated data block according to method 600 includes determining the data sensitivity value of the selected deduplicated data block by evaluating the block data in consideration of the text analysis learning model, and classifying the selected deduplicated data block as sensitive in response to determining that the data sensitivity value of the selected deduplicated data block exceeds a sensitive information threshold.

[0078] Figure 7A method 700 for determining the data sensitivity value of a selected deduplicated data block is illustrated. Method 700 provides one or more embodiments of step 605 of method 600. Method 700 begins at step 705, wherein a data processing application determines corresponding data sensitivity values ​​for multiple portions of the selected deduplicated data block by evaluating data in multiple portions of the block, taking into account a text analytics learning model. In an embodiment, the corresponding data sensitivity values ​​for the multiple block portions are quantitative. According to this embodiment, the corresponding quantitative data sensitivity values ​​for the multiple block portions may optionally be normalized, and / or optionally on a predefined scale, e.g., between 0 and 1, where 0 indicates the lowest sensitivity and 1 indicates the highest sensitivity. In step 710, the data processing application calculates the data sensitivity value of the selected deduplicated data block by aggregating the corresponding data sensitivity values ​​of the multiple portions of the selected deduplicated data block as determined in step 705. In an embodiment, the data processing application aggregates the corresponding data sensitivity values ​​by summing the determined corresponding data sensitivity values ​​of the multiple block portions. In an additional embodiment, the data processing application aggregates the corresponding data sensitivity values ​​by averaging the determined corresponding data sensitivity values ​​of multiple block portions.

[0079] In an embodiment, the data processing application may optionally classify sensitivity at the block portion level, such that the data processing application classifies any portion of the selected deduplicated data block that has a corresponding data sensitivity value exceeding a sensitivity information threshold as sensitive. In the context of classifying one or more corresponding portions of the selected deduplicated data block as sensitive, the data processing application may optionally label or otherwise mark such one or more corresponding portions, for example, in the previously described reference list. In an additional embodiment, in the context of applying at least one result of text analysis according to step 425 to any document in an unstructured document set that includes the selected deduplicated data block, the data processing application classifies any document, or at least a portion thereof, that includes one or more corresponding portions of the selected deduplicated data block that are classified as sensitive as sensitive. In a further embodiment, the data processing application implements access control for any document, or at least a portion thereof, that includes one or more corresponding portions of the selected deduplicated data block that are classified as sensitive.

[0080] In summary, determining the data sensitivity value of the selected deduplicated data block according to method 700 includes: determining the corresponding data sensitivity values ​​of multiple parts of the selected deduplicated data block by taking into account the evaluation of partial data by the text analysis learning model, and calculating the data sensitivity value of the selected deduplicated data block by aggregating the corresponding data sensitivity values ​​of multiple parts of the selected deduplicated data block.

[0081] Figure 8A method 800 for configuring a text analytics learning model is illustrated. Method 800 begins at step 805, wherein a data processing application samples text analytics results from a plurality of previously processed unstructured document sets. In an embodiment, the data processing application predetermines the sampling granularity of the text analytics results from the plurality of previously processed unstructured document sets. According to this embodiment, the data processing application predetermines any one or more types of text analytics results to be sampled, and additionally predetermines which document types or document portion types in the previously processed unstructured document sets to be sampled. According to this embodiment, the data processing application predetermines the sampling granularity based at least in part on input obtained from at least one external entity (e.g., a data processing system administrator or a data processing system client).

[0082] In step 810, the data processing application archives sensitive data based on the sampled text analysis results. In one embodiment, the data processing application archives sensitive data by identifying sensitive data within the sampled text analysis results based on direct text comparison and / or text pattern comparison between the archived sensitive data and the sampled text analysis results. In an additional embodiment, the data processing application identifies sensitive data within the sampled text analysis results via the application of NLP (e.g., NLU). According to one or both embodiments, when identifying sensitive data within the sampled text analysis results, the data processing application archives the identified sensitive data by facilitating the organization (e.g., indexing) of the archived sensitive data based on personal data categories and / or based on confidential data categories within at least one knowledge base associated with the text analysis learning model.

[0083] In one embodiment, a data processing application archives sensitive data based on sampled text analysis results by deriving at least one sensitive data pattern based on the sampled text analysis results. According to this embodiment, the data processing application applies NLP to the sampled text analysis results to derive at least one sensitive data pattern. According to this embodiment, the NLP applied to the sampled text analysis results includes NLU. Optionally, the at least one sensitive data pattern includes at least one n-gram pattern associated with personal data. For example, a sensitive data pattern among the at least one sensitive data pattern may optionally include a corresponding n-gram pattern associated with an individual's birthday. Additionally or alternatively, the at least one sensitive data pattern includes at least one n-gram pattern associated with confidential data. For example, a sensitive data pattern may optionally include an n-gram pattern associated with passwords or other authentication data. In a related embodiment, the data processing application applies contextual analysis to the sampled text analysis results to derive at least one sensitive data pattern. In a further related embodiment, the data processing application applies logical relationship analysis to the sampled text analysis results to derive at least one sensitive data pattern.

[0084] In step 815, the data processing application facilitates training a text analysis learning model based on the archived sensitive data. The data processing application facilitates model training based on the archived sensitive data by decomposing the archived sensitive data into any text analysis algorithms associated with the model. Therefore, the data processing application facilitates model training to facilitate the evaluation of deduplicated data blocks. According to an embodiment in which the data processing application derives at least one sensitive data pattern based on the sampled text analysis results, the data processing application facilitates model training based on at least one sensitive data pattern. In an embodiment, the data processing system updates at least one knowledge base associated with the model based on model training and / or other configuration activities. In an additional embodiment, the data processing system reconfigures the model by performing steps of method 800 while performing steps of method 400 for a corresponding unstructured document set.

[0085] In summary, configuring a text analysis learning model according to Method 800 includes: sampling text analysis results from multiple previously processed unstructured document collections, archiving sensitive data based on the sampled text analysis results, and facilitating the training of the text analysis learning model based on the archived sensitive data.

[0086] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. All kinds of modifications to the described embodiments and equivalent arrangements should fall within the scope of this invention. Therefore, the scope of the invention should be interpreted most broadly according to the following claims relating to the specific embodiments, and should cover all possible equivalent variations and arrangements. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements superior to those found in the market, or to enable those skilled in the art to understand the embodiments described herein.

Claims

1. A computer-implemented method for processing unstructured documents, comprising: Identify multiple deduplicated data blocks associated with a collection of unstructured documents; Based on at least one block frequency metric, the plurality of deduplicated data blocks are sorted in descending order; Select the highest-ranked unprocessed deduplicated data block; Text analysis is applied to the selected deduplicated data block by facilitating the application of at least one natural language processing technique to the selected deduplicated data block; as well as At least one result of the text analysis is applied to any document in the unstructured document set that includes selected deduplicated data blocks, wherein applying the at least one result includes: labeling the document data based on data attributes determined from the text analysis, and characterizing the labeled document data based on the application of at least one machine learning technique.

2. The method according to claim 1, further comprising: The unstructured document processing method is terminated in response to the determination that the unique document used in the next unprocessed deduplicated data block to be selected has a frequency lower than a predetermined document influence threshold.

3. The method according to claim 1, further comprising: The unstructured document processing method is terminated in response to the determination that the frequency of occurrence of a unique block in the next unprocessed deduplicated data block to be selected is lower than a predetermined block occurrence threshold.

4. The method according to claim 1, further comprising: The unstructured document processing method is terminated upon determining that the predetermined evaluation period for unstructured documents has expired.

5. The method of claim 1, further comprising: In response to determining that the number of deduplicated data blocks among the plurality of deduplicated data blocks that have undergone text analysis exceeds a predetermined block text analysis threshold, the unstructured document processing method is terminated.

6. The method of claim 1, further comprising: The unstructured document processing method is terminated in response to the determination that the number of documents in the unstructured document set that have been applied with at least one text analysis result exceeds a predetermined analysis result assignment threshold.

7. The method according to claim 1, wherein, Applying text analysis to selected deduplicated data blocks includes: By taking into account the evaluation of the text analysis learning model of the data block, the data sensitivity value of the selected deduplicated data block is determined.

8. The method according to claim 7, wherein, Applying text analysis to the selected deduplicated data blocks further includes: In response to determining that the data sensitivity value of the selected deduplicated data block exceeds the sensitivity information threshold, the selected deduplicated data block is classified as sensitive.

9. The method according to claim 7, wherein, Determining the data sensitivity value for the selected deduplicated data block includes: By taking into account the evaluation data of the text analysis learning model, the corresponding data sensitivity values ​​for multiple parts of the selected deduplicated data block are determined; and The data sensitivity value of the selected deduplicated data block is calculated by aggregating the corresponding data sensitivity values ​​of the multiple portions of the selected deduplicated data block.

10. The method according to claim 7, wherein, Configuring the text analysis learning model includes: Text analysis results are sampled from multiple previously processed collections of unstructured documents.

11. The method according to claim 10, wherein, Configuring the text analysis learning model further includes: Based on the sampled text analysis results, sensitive data is archived; and This facilitates training the text analysis learning model based on the archived sensitive data.

12. A computer program product comprising unstructured document processing program instructions, the unstructured document processing program instructions being executable by a computing device to cause the computing device to perform the operation of the method according to any one of claims 1 to 11.

13. A computer system, comprising: At least one processor; as well as A memory for storing an application that, when executed on the at least one processor, performs unstructured document processing operations according to any one of claims 1 to 11.

Images