Collaborative neural network with spatial coverage constraints

Abstract

Methods, systems, and computer program products for collaborative neural networks with spatial coverage constraints are provided herein. A computer-implemented method includes partitioning a processing task into a plurality of subtasks; training a plurality of independent neural networks such that at least some of the plurality of subtasks correspond to different ones of the plurality of independent neural networks; defining, based at least in part on constraint-based domain knowledge related to the processing task, at least one constraint loss for a given subtask of the plurality of subtasks, the at least one constraint loss dependent on an output from at least one subtask of the other plurality of subtasks; and retraining at least a portion of the plurality of independent neural networks, the retraining dependent on use of the at least one constraint loss.

Description

Technical Field

[0001] This application relates generally to information technology, and more specifically to data processing technology. Background Technology

[0002] Training neural network models in an end-to-end manner can successfully solve a wide range of tasks. However, end-to-end models typically consume large amounts of data and memory. Furthermore, in many machine learning tasks, the availability of fully labeled data may be limited, while a much larger amount of partially labeled data exists that cannot be effectively utilized by conventional models. Summary of the Invention

[0003] In one embodiment of the invention, techniques are provided for cooperative neural networks having a spatial containment constraint. An exemplary computer-implemented method may include dividing a processing task into multiple subtasks and training multiple independent neural networks such that at least some of the subtasks correspond to different neural networks among the multiple independent neural networks. The method also includes defining at least one constraint loss for a given subtask among the multiple subtasks based at least in part on constraint-based domain knowledge related to the processing task, the at least one constraint loss depending on the output from at least one of the other multiple subtasks, and retraining at least a portion of the multiple independent neural networks, the retraining depending on the use of the at least one constraint loss.

[0004] In another embodiment of the invention, an exemplary computer-implemented method may include: dividing a processing task into a plurality of subtasks; and training a neural network having a plurality of divergent branches therein, such that at least some of the subtasks correspond to different branches of the plurality of divergent branches of the neural network. The method further includes defining at least one constraint loss for a given subtask among the plurality of subtasks based at least in part on constraint-based domain knowledge relating to the processing task, the at least one constraint loss depending on the output from at least one of the other plurality of subtasks; and retraining at least a portion of the neural network, the retraining depending on the use of the at least one constraint loss.

[0005] Another embodiment of the invention or elements thereof can be implemented in the form of a computer program product tangibly embodying computer-readable instructions that, when implemented, cause a computer to perform a plurality of method steps as described herein. Furthermore, another embodiment of the invention or elements thereof can be implemented in the form of a system including a memory and at least one processor coupled to the memory and configured to perform the indicated method steps. Further still, another embodiment of the invention or elements thereof can be implemented in the form of an apparatus or elements thereof for performing the method steps described herein; the apparatus may include hardware modules or a combination of hardware and software modules, wherein the software modules are stored in a tangible computer-readable storage medium (or multiple media).

[0006] These and other objects, features, and advantages of the invention will become apparent from the following detailed description of exemplary embodiments of the invention, which will be read in conjunction with the accompanying drawings. Attached Figure Description

[0007] Figure 1 This is a diagram illustrating the system architecture according to an exemplary embodiment of the present invention;

[0008] Figure 2 This is a flowchart illustrating a technique according to an embodiment of the present invention;

[0009] Figure 3 This is a system diagram of an exemplary computer system on which at least one embodiment of the present invention can be implemented;

[0010] Figure 4 A cloud computing environment according to an embodiment of the present invention is shown; and

[0011] Figure 5 An abstract model layer is described according to an embodiment of the present invention. Detailed Implementation

[0012] As described herein, embodiments of the invention include collaborative neural networks with spatial coverage constraints. More specifically, at least one embodiment includes generating and / or implementing a constraint-based neural network training framework for machine learning tasks. In one or more embodiments, the framework includes implementing the steps of: dividing a given task into multiple subtasks; training at least one independent neural network for each of the respective subtasks (e.g., potentially using different datasets); leveraging constraint-based domain knowledge related to the given task and defining a constraint loss relating to the outcome of each subtask; and sequentially training each neural network model in an adversarial style using predictions from one or more other models.

[0013] Therefore, as emphasized in at least part above, at least one embodiment includes: dividing a given task into multiple subtasks (e.g., detecting tables and cells of those tables separately); training a separate model for each subtask using potentially different datasets; defining and / or learning at least one spatial constraint relationship between the outputs of each subtask; adding at least one loss to each subtask model; penalizing constraint violations and continuing training; and further modifying and / or fine-tuning the framework (e.g., using only constraint losses with unlabeled data).

[0014] This framework, as described in at least one embodiment herein, is also referred to as a Constraint-Based Collaborative Neural Network (C2NN) because multiple neural networks collaborate with each other in accordance with one or more predefined constraints. C2NN offers at least the following advantages over conventional modeling and / or analysis methods. For example, C2NN can augment the use (e.g., full use) of partially labeled data to simultaneously capture robust local features and preserve global structure. Additionally, during the training phase associated with the example embodiment, each model is trained individually, allowing for the use of partially labeled data to train more robust subtask models. As also covered as part of the training phase, multiple neural networks collaborate with each other to preserve the global structure of a given (overall) task.

[0015] Another advantage over conventional methods includes memory efficiency. For example, training a large number of end-to-end models typically requires considerable graphics processing unit (GPU) and / or tensor processing unit (TPU) memory. However, in at least one embodiment, because only one neural network for a subtask is trained in real-time at any given time during C2NN implementation, smaller sub-models can be trained using C2NN in resource-intensive environments. Additionally, in one or more embodiments, C2NN can be used to enhance domain adaptation. For example, during the training phase, C2NN can facilitate adapting the current model to different domains using limited labeled data by preserving the same constraints possessed by both the source and target domains.

[0016] Therefore, and as detailed herein, one or more embodiments include incorporating partially labeled data (e.g., for training purposes). Because backpropagation is efficient for the neurons of the model involved in the training samples, end-to-end models are typically resource-intensive. However, at least one embodiment includes training only a (relatively small) sub-task model at each timestamp.

[0017] Furthermore, at least one embodiment includes achieving and / or facilitating collaboration across multiple neural networks by independently detecting at least one aspect of a problem that subsequently becomes a constraint that other networks can incorporate. This constraint may be related to coverage, for example, such that the output of one network needs to be physically contained within the output of another network.

[0018] As an additional example, by using spatial relationships between sub-objects, one or more embodiments may include: using partially labeled data (which does not contain all the labels required for the complete task), incorporating domain knowledge to improve training speed and accuracy, and learning from unlabeled data.

[0019] Figure 1 An example of a training mechanism with constraint-based loss and table recognition is described. However, one or more embodiments can be implemented and / or applied to any network whose output has some constraints (e.g., coverage constraints, temporal consistency constraints, etc.). For example, at least one embodiment may include an implementation within the context of at least one temporal similarity constraint, wherein two networks that predict future and past events respectively may have similar results, and the pixel-level similarity of the outputs of the two networks becomes a constraint.

[0020] like Figure 1 As shown, the table boundary framework 104 can utilize the unit detection network by taking advantage of the fact that a table must contain at least some units. During the training phase, in addition to regression and classification losses, one or more embodiments include adding a segmented constraint loss that penalizes unrealistic table detection probabilities when considering unit locations. This novel unit constraint-based loss function can be added to any detection network.

[0021] Therefore, such as Figure 1 As depicted, the table boundary frame 104 includes a table object detector 106 and a cell object detector 108. The table object detector 106 detects one or more tables 110 in the input (full-page) image 102, while the cell object detector 108 detects one or more cells 114 in the input (full-page) image 102. Based on the one or more detected tables 110, candidate table bounding boxes 112 are determined. Based on the one or more detected cells 114, one or more cell bounding boxes 116 are determined.

[0022] One or more embodiments include using a guided cell network to generate a set of cell bounding boxes (e.g., cell bounding box 116). Given this set of cell bounding boxes, the embodiment may include defining a Boolean operator, where the input includes inner boxes and outer boxes, wherein the input defines the boundaries of a masked input region. A penalty indicator is true when less than a certain percentage of the table contains cells.

[0023] During the inference phase, one or more embodiments include ranking the proposed bounding boxes by considering the presence of cells outside and inside the table and the detection probability. The embodiment may also include defining constraint coefficients for each bounding box and, for each boundary of a table bounding box, calculating the number of cells outside the table minus the number of cells inside the table. For any pair of bounding boxes that overlaps with each other by more than a given percentage, the embodiment includes discarding the bounding box with the higher constraint coefficient.

[0024] Referring to the table recognition example above, the fact that cells must be located inside tables represents a spatial constraint between tables and cells. Using conventional modeling methods, independent neural networks detecting each aspect (cells and tables) cannot take advantage of this type of global constraint. However, as detailed in the example use cases above, one or more embodiments may include leveraging the outputs of each individual model and / or network, as well as domain knowledge of how cells and tables relate, to improve detection during both the training and inference phases.

[0025] Additional example use cases for implementing one or more embodiments of the present invention may include zero-shot learning. For example, when detecting a new animal type (such as a zebra), there may be a network pre-trained as a horse detector and another network pre-trained as a stripe detector. Since zebras are known to resemble horses in some respects but have stripes, a penalty can be added to the horse detector so that any detection that is activated even if it is not a stripe detector within the animal category is penalized. Similarly, any stripe detector that is not within the horse detector category can also be penalized. Ultimately, the network can be adapted to detect zebras even when there are no labeled zebra instances.

[0026] Figure 2 This is a flowchart illustrating a technique according to an embodiment of the present invention. Step 202 includes dividing the processing task into multiple subtasks. The multiple subtasks may correspond to different parts of a given object and / or at least a portion of another subtask among the multiple subtasks. Furthermore, in at least one embodiment, the processing task includes determining the structure and location of one or more tables in a document. In this embodiment, coverage constraints may exist between cells and table boundaries for table identification. For example, cells should always be contained within tables, and tables should always contain cells. Therefore, when a cell is detected outside any table detected by the table network, additional penalties are added to the cell network for the detection of additional cells. Additional penalties are added to the table network if a table is detected where the density of detected cells is low, or if several cells are detected outside of that table.

[0027] Step 204 includes training multiple independent neural networks such that at least some of the multiple subtasks correspond to different neural networks among the multiple independent neural networks. In at least one embodiment, training multiple independent neural networks may include training multiple independent neural networks such that each network is specialized for detecting at least two types of objects, wherein the at least two types of objects are known to have at least one spatial relationship with each other (e.g., an overlay relationship).

[0028] Step 206 includes defining at least one constraint loss for a given subtask among a plurality of subtasks, based at least in part on constraint-based domain knowledge related to the processing task, the at least one constraint loss depending on the output from at least one of the other subtasks. In at least one embodiment, defining at least one constraint loss includes automatically discovering the corresponding constraints by analyzing labeled data.

[0029] Step 208 includes retraining at least a portion of a plurality of independent neural networks, the retraining depending on the use of at least one constraint loss. In at least one embodiment, given that for any given input, at least one spatial relationship is violated between one or more outputs of at least one neural network and at least a portion of other neural networks, the retraining includes incorporating at least one constraint loss into at least one of the plurality of independent neural networks. This embodiment may also include backpropagating at least one constraint loss through at least one neural network by modifying the weights assigned to each of one or more units in at least one neural network toward values ​​that will produce outputs that satisfy the constraints. Additionally or alternatively, in one or more embodiments, the retraining includes using at least one constraint loss in a supervised environment or in an unsupervised environment.

[0030] Furthermore, another embodiment of the invention includes dividing a processing task into multiple subtasks, and training a neural network having multiple divergent branches therein, such that at least some of the subtasks correspond to different branches of the multiple divergent branches of the neural network. This embodiment also includes defining at least one constraint loss for a given subtask among the multiple subtasks, at least in part based on constraint-based domain knowledge related to the processing task, the at least one constraint loss depending on the output from at least one of the other multiple subtasks, and retraining at least a portion of the neural network, the retraining depending on the use of at least one constraint loss.

[0031] As described in this article, Figure 2The techniques described herein may also include providing a system comprising different software modules, each of which is embodied on a tangible computer-readable and recordable storage medium. For example, all modules (or any subset thereof) may be on the same medium, or each may be on a different medium. Modules may include any or all of the components shown in the figures and / or described herein. In embodiments of the invention, modules may, for example, run on a hardware processor. The different software modules of the system described above, executed on a hardware processor, can then be used to perform method steps. Further, the computer program product may include a tangible computer-readable and recordable storage medium having code adapted to perform at least one method step described herein, the method comprising providing different software modules to the system.

[0032] Alternatively, it can be achieved through computer program products. Figure 2 The technology described herein includes a computer program product that may include computer-usable program code stored in a computer-readable storage medium within a data processing system, and wherein the computer-usable program code is downloaded from a remote data processing system via a network. Furthermore, in embodiments of the invention, the computer program product may include computer-usable program code stored in a computer-readable storage medium within a server data processing system, and wherein the computer-usable program code is downloaded to a remote data processing system via a network for use in a computer-readable storage medium having a remote system.

[0033] Embodiments of the present invention or elements thereof can be implemented in the form of a device including a memory and at least one processor, wherein the at least one processor is coupled to the memory and configured to perform exemplary method steps.

[0034] Furthermore, embodiments of the present invention can utilize software running on a computer or workstation. See also Figure 3This implementation may employ, for example, a processor 302, a memory 304, and an input / output interface, such as a display 306 and a keyboard 308. As used herein, the term "processor" is intended to include any processing device, such as, for example, a processing device including a CPU (Central Processing Unit) and / or other forms of processing circuitry. Furthermore, the term "processor" may refer to more than one separate processor. The term "memory" is intended to include memory associated with a processor or CPU, such as RAM (Random Access Memory), ROM (Read-Only Memory), fixed storage devices (e.g., hard disk drives), removable storage devices (e.g., disk drives), flash memory, etc. Furthermore, as used herein, the phrase "input / output interface" is intended to include, for example, mechanisms for inputting data into the processing unit (e.g., a mouse) and mechanisms for providing results associated with the processing unit (e.g., a printer). The processor 302, memory 304, and input / output interfaces such as the display 306 and keyboard 308 may be interconnected, for example, via a bus 310 that is part of a data processing unit 312. Suitable interconnection (e.g., via bus 310) may also be provided for network interface 314 (such as a network card) and media interface 316 (such as a disk or CD-ROM drive), providing network interface 314 for interface connection with a computer network and media interface 316 for interface connection with media 318.

[0035] Therefore, computer software comprising instructions or code for performing the methods of the present invention as described herein can be stored in an associated memory device (e.g., ROM, fixed or removable memory), and when the computer software is ready for use, part or all of the computer software is loaded (e.g., loaded into RAM) and implemented by the CPU. This software may include, but is not limited to, firmware, resident software, microcode, etc.

[0036] A data processing system suitable for storing and / or executing program code may include at least a processor 302, which is directly or indirectly coupled to a memory element 304 via a system bus 310. The memory element may include local memory used during actual implementation of the program code, mass storage, and a cache memory that provides temporary storage for at least some of the program code to reduce the number of times code must be retrieved from the mass storage during implementation.

[0037] Input / output or I / O devices (including but not limited to keyboard 308, display 306, pointing devices, etc.) can be coupled to the system directly (such as via bus 310) or through an intermediate I / O controller (omitted for clarity).

[0038] Network adapters such as network interface 314 can also be coupled to the system, enabling the data processing system to couple to other data processing systems or remote printers or storage devices via an intermediate private or public network. Modems, cable modems, and Ethernet cards are just some of the types of network adapters currently available.

[0039] As used herein, including in the claims, "server" includes a physical data processing system that runs server programs (e.g., such as...). Figure 3 (System 312 shown). It will be understood that the physical server may or may not include a monitor and keyboard.

[0040] This invention can be a system, method, and / or computer program product with any possible level of technical detail. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0041] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punched cards, or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0042] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0043] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may be personalized to execute computer-readable program instructions by utilizing state information from the computer-readable program instructions in order to perform aspects of this invention.

[0044] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0045] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in the boxes or blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing containing instructions that implement aspects of the functions / actions specified in the boxes or blocks of a flowchart and / or block diagram.

[0046] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0047] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a portion, segment, or module of instructions, comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two consecutive blocks shown may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0048] It should be noted that any method described herein may include additional steps of providing a system comprising different software modules embodied on a computer-readable storage medium; these modules may include any or all of the components detailed herein, for example. The method steps may then be performed using the different software modules and / or submodules of the system as described above, which execute on hardware processor 302. Furthermore, the computer program product may include a computer-readable storage medium having code adapted to be implemented to perform at least one method step described herein, the method comprising providing these different software modules to the system.

[0049] In any case, it will be understood that the components shown herein can be implemented in various forms, such as hardware, software, or combinations thereof, for example, application-specific integrated circuits (ASICs), functional circuits, appropriately programmed digital computers with associated memory, etc. Given the teachings of the invention provided herein, those skilled in the art will be able to conceive of other implementations of the components of the invention.

[0050] Furthermore, it should be understood in advance that the implementation of the teachings cited herein is not limited to a specific computing environment. Rather, embodiments of the invention can be implemented in conjunction with any type of computing environment now known or developed hereafter.

[0051] For example, cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0052] The features are as follows:

[0053] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.

[0054] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0055] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0056] Rapid flexibility: Capacity can be provided quickly and flexibly, automatically scaling down and up rapidly in some cases to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0057] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0058] The service model is as follows:

[0059] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. The applications can be accessed from different client devices via a thin client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0060] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using tools and programming languages ​​supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.

[0061] Infrastructure as a Service (IaaS): This provides consumers with the capability to deliver processing, storage, networking, and other basic computing resources that enable them to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0062] The deployment model is as follows:

[0063] Private cloud: The cloud infrastructure is solely for the organization's operation. This cloud infrastructure can be managed by the organization or a third party and can exist on-site or off-site.

[0064] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). This cloud infrastructure can be managed by the organization or a third party and can exist on-site or off-site.

[0065] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by the organization that sells cloud services.

[0066] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0067] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.

[0068] See now Figure 4 The diagram illustrates an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate. These local computing devices include, for example, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service, without requiring cloud consumers to maintain resources on their local computing devices for this service. It is understood that... Figure 4The types of computing devices 54A-N shown are intended to be illustrative only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connectivity (e.g., using a web browser).

[0069] See now Figure 5 This demonstrates a cloud computing environment of 50 ( Figure 4 This provides a set of functional abstractions. These can be understood in advance. Figure 5 The components, layers, and functions shown are merely illustrative, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided:

[0070] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0071] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75. In one example, the management layer 80 can provide the functionality described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources.

[0072] In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. The user portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides the allocation and management of cloud computing resources to meet the required service levels. Service level agreement (SLA) planning and fulfillment 85 provides the pre-scheduling and procurement of cloud computing resources based on anticipated future needs according to the SLA.

[0073] Workload layer 90 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: map creation and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis and processing 94; transaction processing 95; and collaborative neural networks with spatial coverage constraints according to one or more embodiments of the present invention 96.

[0074] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to also include the plural forms. It is also understood that when the terms “comprising” and / or “including” are used in this specification, they specify the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of another feature, step, operation, element, component, and / or combination thereof.

[0075] At least one embodiment of the present invention can provide beneficial effects, such as effectively utilizing partially labeled data that does not contain all the labels required for the entire task, combining domain knowledge to improve network training speed and accuracy, and learning from unlabeled data.

[0076] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method, comprising: The processing task is divided into multiple sub-tasks, wherein the processing task includes detecting objects or determining the structure or location of one or more tables in a document; Train multiple independent neural networks such that at least some of the multiple subtasks correspond to different neural networks among the multiple independent neural networks, wherein each of the multiple independent neural networks is trained to specifically detect at least two types of objects, wherein the at least two types of objects have at least one spatial relationship with each other; At least in part, based on constraint-based domain knowledge related to the processing task, define at least one constraint loss for a given subtask among the plurality of subtasks, the at least one constraint loss depending on the output from at least one of the other plurality of subtasks; and Retraining at least a portion of the plurality of independent neural networks, the retraining depending on the use of the at least one constraint loss, wherein, given that for any given input, one or more outputs of the at least one neural network violate the at least one spatial relationship between at least a portion of other neural networks, the retraining includes incorporating the at least one constraint loss into at least one of the plurality of independent neural networks; The method is performed by at least one computing device.

2. The computer-implemented method according to claim 1, wherein, The multiple subtasks correspond to different parts of a given object.

3. The computer-implemented method according to claim 1, wherein, One of the plurality of subtasks corresponds to at least a portion of another of the plurality of subtasks.

4. The computer-implemented method according to claim 1, wherein, The at least one spatial relationship includes an enclosure relationship.

5. The computer-implemented method according to claim 1, comprising: The at least one constraint loss is backpropagated through the at least one neural network by modifying the weights assigned to each of one or more units in the at least one neural network toward values ​​that will produce an output that satisfies the constraints.

6. The computer-implemented method according to claim 1, wherein, The definition of at least one constraint loss includes automatically discovering the corresponding constraint through analysis of the labeled data.

7. The computer-implemented method according to claim 1, wherein, The retraining includes using at least one of the constraint losses in a supervised setting.

8. The computer-implemented method according to claim 1, wherein, The retraining includes using at least one of the constraint losses in an unsupervised setting.

9. A computer program product comprising program instructions executable by a computing device to cause the computing device to: The processing task is divided into multiple subtasks, among which... The processing task includes detecting objects or determining the structure or location of one or more tables in a document; Train multiple independent neural networks such that at least some of the multiple subtasks correspond to different neural networks among the multiple independent neural networks, wherein each of the multiple independent neural networks is trained to specifically detect at least two types of objects, wherein the at least two types of objects have at least one spatial relationship with each other; At least in part, based on constraint-based domain knowledge related to the processing task, define at least one constraint loss for a given subtask among the plurality of subtasks, the at least one constraint loss depending on the output from at least one of the other plurality of subtasks; and Retraining at least a portion of the plurality of independent neural networks, the retraining depending on the use of the at least one constraint loss, wherein, given that for any given input, one or more outputs of the at least one neural network violate the at least one spatial relationship between the at least one portion of other neural networks, the retraining includes incorporating the at least one constraint loss into the at least one of the plurality of independent neural networks.

10. A computer system, comprising: Memory; as well as At least one processor, operatively coupled to the memory and configured to: The processing task is divided into multiple sub-tasks, wherein the processing task includes detecting objects or determining the structure or location of one or more tables in a document; Train multiple independent neural networks such that at least some of the multiple subtasks correspond to different neural networks among the multiple independent neural networks, wherein each of the multiple independent neural networks is trained to specifically detect at least two types of objects, wherein the at least two types of objects have at least one spatial relationship with each other; At least in part, based on constraint-based domain knowledge related to the processing task, define at least one constraint loss for a given subtask among the plurality of subtasks, the at least one constraint loss depending on the output from at least one of the other plurality of subtasks; and Retraining at least a portion of the plurality of independent neural networks, the retraining depending on the use of the at least one constraint loss, wherein, given that for any given input, one or more outputs of the at least one neural network violate the at least one spatial relationship between the at least one portion of other neural networks, the retraining includes incorporating the at least one constraint loss into the at least one of the plurality of independent neural networks.

11. A computer-implemented method, comprising: The processing task is divided into multiple sub-tasks, wherein the processing task includes detecting objects or determining the structure or location of one or more tables in a document; A neural network having multiple divergent branches is trained such that at least some of the multiple subtasks correspond to different branches of the multiple divergent branches of the neural network, wherein the neural network is trained to detect at least two types of objects, wherein the at least two types of objects have at least one spatial relationship with each other. At least in part, based on constraint-based domain knowledge related to the processing task, define at least one constraint loss for a given subtask among the plurality of subtasks, the at least one constraint loss depending on the output from at least one of the other plurality of subtasks; and Retraining at least a portion of the neural network, the retraining depending on using the at least one constraint loss, wherein, given that for any given input, one or more outputs of the at least one neural network violate the at least one spatial relationship between at least a portion of other neural networks, the retraining includes incorporating the at least one constraint loss into the neural network; The method is performed by at least one computing device.

12. The computer-implemented method according to claim 11, wherein, The multiple subtasks correspond to different parts of a given object.

13. The computer-implemented method according to claim 11, wherein, One of the plurality of subtasks corresponds to at least a portion of another of the plurality of subtasks.

14. The computer-implemented method according to claim 11, comprising: The at least one constraint loss is backpropagated through the at least one neural network by modifying the weights assigned to each of one or more units in the at least one neural network toward values ​​that will produce an output that satisfies the constraints.

Images