Cooperative Neural Networks with Spatial Confinement Constraints

Abstract

Provided herein are methods, systems, and computer program products for cooperative neural networks with spatial confinement constraints. The computer-implemented methods include dividing a processing task into a plurality of sub-tasks, training a plurality of independent neural networks such that at least some of the plurality of sub-tasks correspond to different ones of the plurality of independent neural networks, determining, for a given one of the plurality of sub-tasks based at least in part on constraint-based domain knowledge related to the processing task, at least one constraint loss that depends on outputs from at least one of the other plurality of sub-tasks, and retraining at least some of the plurality of independent neural networks using the at least one constraint loss.

Description

[Technical field]

[0001] TECHNICAL FIELD This application relates generally to information technology, and more specifically to data processing technology. [Background technology]

[0002] Various tasks can be successfully solved by training neural network models end-to-end. However, end-to-end models usually consume large amounts of data and memory. Moreover, for many machine learning tasks, the availability of fully labeled data may be limited, while there is a large amount of partially labeled data that cannot be efficiently utilized by traditional models. Summary of the Invention

[0003] In one embodiment of the present invention, techniques are provided for cooperative neural networks with spatial containment constraints. An exemplary computer-implemented method includes dividing a processing task into a plurality of sub-tasks and training a plurality of independent neural networks, such that at least some of the plurality of sub-tasks correspond to different ones of the plurality of independent neural networks. The method further includes determining, for a given one of the plurality of sub-tasks based at least in part on constraint-based domain knowledge regarding the processing task, at least one constraint loss that depends on outputs from at least one of the other plurality of sub-tasks, and retraining at least some of the plurality of independent neural networks in dependence on using the at least one constraint loss.

[0004] In another embodiment of the invention, an exemplary computer-implemented method includes dividing a processing task into a plurality of sub-tasks and training a neural network having a plurality of branching branches such that at least some of the plurality of sub-tasks correspond to different ones of the plurality of branching branches of the neural network. The method further includes determining, for a given one of the plurality of sub-tasks based at least in part on constraint-based domain knowledge regarding the processing task, at least one constraint loss that depends on output from at least one of the other plurality of sub-tasks, and retraining at least a portion of the neural network in response to using the at least one constraint loss.

[0005] Another embodiment of the invention or elements thereof may be implemented in the form of a computer program product tangibly embodying computer-readable instructions that, when implemented, cause a computer to perform a number of the method steps described herein. Still further, another embodiment of the invention or elements thereof may be implemented in the form of a system including a memory and at least one processor coupled to the memory and configured to perform said method steps. Still further, another embodiment of the invention or elements thereof may be implemented in the form of a means or elements thereof for performing the method steps described herein, which means may include a hardware module(s) or a combination of hardware and software modules, which software modules are stored in a tangible computer-readable storage medium(s).

[0006] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates a system architecture according to an exemplary embodiment of the present invention. [Diagram 2] FIG. 1 is a flow diagram illustrating a technique according to one embodiment of the present invention. [Diagram 3] FIG. 1 is a system diagram of an exemplary computer system in which at least one embodiment of the present invention may be implemented. [Figure 4] 1 illustrates a cloud computing environment according to one embodiment of the present invention. [Diagram 5] 3 illustrates abstraction model layers according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] As described herein, an embodiment of the present invention includes a cooperative neural network with a spatial confinement constraint. More specifically, at least one embodiment includes generating and / or implementing a constraint-based neural network training framework for machine learning tasks. The framework, in one or more embodiments, includes the following steps: splitting a given task into multiple sub-tasks, training at least one independent neural network for each respective sub-task (e.g., with different possible datasets), utilizing constraint-based domain knowledge about the given task to define a constraint loss that includes the results of each sub-task, and adversarially training each neural network model in turn with the prediction results of one or more other models.

[0009] Thus, as at least partially highlighted above, at least one embodiment includes splitting a given task into multiple sub-tasks (e.g., detecting tables and cells in those tables separately), training a separate model for each sub-task on potentially different datasets, defining and / or learning spatial constraint relationships between the outputs of each sub-task, adding at least one loss to each sub-task model, continuing training with a penalty for constraint violations, and / or further modifying or fine-tuning the framework (e.g., with only constraint losses on unlabeled data).

[0010] Such a framework of at least one embodiment is also referred to herein as a constraint-based collaborative neural network (C2NN), in which multiple neural networks cooperate with each other in relation to one or more predefined constraints. C2NN provides at least the following advantages over traditional modeling and / or analysis approaches. For example, C2NN can increase (e.g., fully) use partially labeled data to capture robust local features and simultaneously maintain global structure. Furthermore, during the training phase associated with an exemplary embodiment, each model is trained separately so that more robust sub-task models can be trained using partially labeled data. Also included as part of the training phase are multiple neural networks that cooperate with each other to maintain the global structure of a given (overall) task.

[0011] Another advantage over conventional approaches includes memory efficiency. For example, training a substantial end-to-end model often requires a large amount of memory on the graph processing unit (GPU) or tensor processing unit (TPU) or both. However, in at least one embodiment, since in any C2NN implementation, only one neural network is trained on the fly for a sub-task, C2NN can be used to train smaller sub-models in resource-intensive environments. Additionally, in one or more embodiments, C2NN can be used to enhance domain adaptability. For example, during the training phase, C2NN can facilitate the adaptation of the current model to a different domain with limited labeled data by maintaining the same constraints possessed by both the source and target domains.

[0012] Thus, and as detailed herein, one or more embodiments include incorporating partially labeled data (e.g., for training purposes). End-to-end models are often resource intensive, due to the back-propagation that is effective for the model's neurons contained in the training samples. However, at least one embodiment includes training only a (relatively small) sub-task model at each time stamp.

[0013] Additionally, at least one embodiment includes implementing and / or facilitating coordination across multiple neural networks by independently detecting at least one aspect of a problem that results in at least one constraint that the other network(s) can incorporate. By way of example only, such constraints may relate to containment, such as requiring that the output of one network be physically contained within the output of another network.

[0014] As an additional example, by using spatial relationships between objects, one or more embodiments can include using partially labeled data that does not include all labels required by the entire task, incorporating domain knowledge to improve training speed and accuracy, and learning from unlabeled data.

[0015] FIG. 1 shows an example of a constraint loss based training scheme with table recognition. However, one or more embodiments can be implemented and / or applied to any network whose outputs have some constraint relationship (e.g., confinement constraint, temporal consistency constraint, etc.). For example, at least one embodiment can include implementation within the context of at least one temporal similarity constraint, where two networks, each predicting future and past events, are likely to have similar results, and pixel-level similarity of their outputs is a constraint.

[0016] As shown in Figure 1, the table boundary framework 104 can employ cell detection networks by taking advantage of the fact that a table must contain at least some cells. In the training phase, in addition to regression and classification losses, one or more embodiments include adding a piecewise constrained loss that penalizes the probability of detecting unrealistic tables when considering cell positions. This new cell constraint-based loss function can be added to any detection network.

[0017] 1, the table boundary framework 104 includes an object detector for tables 106 and an object detector for cells 108. The object detector for tables 106 detects one or more tables 110 in the input (full page) image 102, while the object detector for cells 108 detects one or more cells 114 in the input (full page) image 102. From the one or more detected tables 110, candidate table bounding boxes 112 are determined. From the one or more detected cells 114, one or more cell bounding boxes 116 are determined.

[0018] One or more embodiments include using a guided cell network to generate a set of cell bounding boxes (e.g., cell bounding box 116). Given the set of cell bounding boxes, such embodiments can include defining a Boolean operator whose inputs include an inner box and an outer box, which define a boundary for a mask input region. A penalty indication is true when the table has fewer than a certain percentage of cells.

[0019] In the estimation stage, one or more embodiments include ranking the proposed bounding boxes by considering the detection probability and the presence of cells inside and outside the table. Such embodiments further include determining a constraint factor for each bounding box and a constraint factor for each boundary of the table bounding box, and performing a calculation of the number of cells outside the table minus the number of cells inside the table. Such embodiments include, for any pair of bounding boxes that overlap each other by more than a given percentage, discarding the bounding box with the higher constraint factor.

[0020] In relation to the above example of table recognition, the fact that a cell must be inside a table represents a spatial constraint between the table and the cell. Using traditional modeling approaches, independent neural networks detecting each aspect (cell and table) cannot take advantage of this type of global constraint. However, as detailed in the exemplary use cases above, one or more embodiments can take advantage of the output of each individual model or network or both, and domain knowledge of how cells and tables relate, to improve detection in both the training and inference phases.

[0021] Additional exemplary use cases for implementing one or more embodiments of the present invention may include zero-shot learning. For example, when detecting a new type of animal, such as a zebra, one may have a network pre-trained as a horse detector and another network pre-trained as a stripe detector. Since zebras are similar to horses in some ways but have stripes, the horse detector may be penalized such that any detections without the stripe detector working within the animal are penalized. Similarly, any stripe detector without the horse detector may also be penalized. Finally, the network may be adapted to detect zebras even when there are no labeled examples of zebras at all.

[0022] FIG. 2 is a flow diagram illustrating a technique according to an embodiment of the present invention. Step 202 includes splitting a processing task into multiple sub-tasks. The multiple sub-tasks can correspond to different parts of a given object, or at least a part of another one of the multiple sub-tasks, or both. In at least one embodiment, the processing task also includes determining the structure and location of one or more tables in the document. In such an embodiment, the containment constraints may be on the cell and table boundaries for table recognition. For example, a cell should always be contained inside a table, and a table should always contain a cell. Thus, an extra penalty is added to the cell network for extra cell detection when a cell is detected outside any table detected by the table network. An extra penalty is added to the table network when a table is detected with a low density of cells detected within the table, or when there are some cells detected just outside the table.

[0023] Step 204 includes training a plurality of independent neural networks such that at least some of the plurality of sub-tasks correspond to different ones of the plurality of independent neural networks. In at least one embodiment, training the plurality of independent neural networks includes training the plurality of independent neural networks such that each network is specialized to detect at least two types of objects that are known to have at least one spatial relationship (e.g., a containment relationship) with one another.

[0024] Step 206 determines, for a given one of the plurality of sub-tasks, at least one constrained loss that depends on outputs from at least one of the other plurality of sub-tasks based at least in part on constraint-based domain knowledge regarding the processing task. In at least one embodiment, determining the at least one constrained loss includes automatically discovering corresponding constraints through analysis of the labeled data.

[0025] Step 208 retrains at least some of the multiple independent neural networks in dependence on using at least one constraint loss. In at least one embodiment, the retraining includes incorporating at least one constraint loss into at least one of the multiple independent neural networks considering that for any given input, there is a violation of at least one spatial relationship between one or more outputs of the at least one neural network and one or more outputs of at least some of the other neural networks. Such an embodiment further includes backpropagating the at least one constraint loss throughout the at least one neural network by modifying weights assigned to each of one or more units in the at least one neural network toward values ​​that would have produced an output that satisfies the constraint. Additionally or alternatively, in one or more embodiments, the retraining includes using the at least one constraint loss in a supervised or unsupervised setting.

[0026] Yet additional embodiments of the present invention include dividing a processing task into a plurality of sub-tasks and training a neural network having a plurality of branching branches such that at least some of the plurality of sub-tasks correspond to different ones of the plurality of branching branches of the neural network. Such embodiments further include determining, for a given one of the plurality of sub-tasks based at least in part on constraint-based domain knowledge regarding the processing task, at least one constraint loss that depends on output from at least one of the other plurality of sub-tasks, and retraining at least a portion of the neural network, where the retraining depends on using the at least one constraint loss.

[0027] The technique shown in FIG. 2 also includes preparing a system including separate software modules, each embodied on a tangible computer-readable recordable storage medium, as described herein. All of the modules (or any subset thereof) can be on the same medium, or each can be on, for example, different media. The modules can include any or all of the components shown in the figure and / or described herein. In one embodiment of the invention, the modules can be executed, for example, on a hardware processor. Method steps are then performed using the separate software modules of the system, as described above, executed on the hardware processor. Additionally, the computer program product can include a tangible computer-readable recordable storage medium having code adapted to be executed to perform at least one method step as described herein, including preparing a system having separate software modules.

[0028] 2 may be implemented via a computer program product that may include computer usable program code stored in a computer readable storage medium in a data processing system, where the computer usable program code is downloaded over a network from a remote data processing system. Additionally, in one embodiment of the present invention, the computer program product may include computer usable program code stored in a computer readable storage medium in a server data processing system, where the computer usable program code is downloaded over a network to the remote data processing system for use in the computer readable storage medium by the remote system.

[0029] An embodiment of the present invention, or elements thereof, may be embodied in the form of an apparatus including a memory and at least one processor coupled to the memory and configured to perform the exemplary method steps.

[0030] Further, an embodiment of the present invention may employ software executed on a computer or workstation. With reference to FIG. 3, such an implementation may employ, for example, a processor 302, a memory 304, and an input / output interface formed, for example, by a display 306 and a keyboard 308. The term "processor" as used herein is intended to include any processing device, such as a processing device including a CPU (Central Processing Unit) or other forms of processing circuitry or both. Furthermore, the term "processor" may refer to one or more individual processors. The term "memory" is intended to include memory associated with a processor or CPU, such as RAM (Random Access Memory), ROM (Read Only Memory), fixed memory devices (e.g., hard drives), removable memory devices (e.g., diskettes), flash memory, and the like. Furthermore, the phrase "input / output interface" as used herein is intended to include, for example, mechanisms for inputting data into a processing unit (e.g., a mouse) and mechanisms for providing results associated with a processing unit (e.g., a printer). The processor 302, memory 304, and input / output interfaces such as a display 306 and a keyboard 308 may be interconnected, for example, via a bus 310 as part of a data processing system 312. For example, suitable interconnections via the bus 310 may include a network interface 314, such as a network card, that may provide an interface with a computer network, and a media interface 316, such as a diskette or CD-ROM drive, that may provide an interface with a media 318.

[0031] Thus, computer software containing instructions or code for performing the methods of the present invention, as described herein, can be stored in associated memory devices (e.g., ROM, fixed or removable memory) and, when ready for use, partially or fully loaded (e.g., into RAM) and executed by the CPU. Such software can include, but is not limited to, firmware, resident software, microcode, etc.

[0032] A data processing system suitable for storing and / or executing program code will include at least one processor 302 coupled directly to memory elements 304 or indirectly through a system bus 310. The memory elements may include local memory used during execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some of the program code to reduce the number of times the code needs to be retrieved from bulk storage during execution.

[0033] Input / output or I / O devices (including but not limited to keyboard 308, display 306, pointing device, etc.) may be coupled to the system either directly (such as via bus 310) or through intervening I / O controllers (omitted for clarity).

[0034] Network adapters, such as network interface 314, may also be connected to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems and Ethernet cards are some of the currently available types of network adapters.

[0035] As used herein, including in the claims, a "server" includes a physical data processing system that executes a server program (e.g., system 312 shown in FIG. 3). It should be understood that such a physical server may or may not include a display and keyboard.

[0036] The present invention may be embodied as a system, method, or computer program product, or combination thereof, at any possible level of technical detail. The computer program product may include a computer readable storage medium or media having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0037] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves having instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium is not to be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or an electrical signal sent through a wire.

[0038] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or combinations thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions for storage in a computer readable storage medium in the respective computing / processing device.

[0039] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the last scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the invention.

[0040] Aspects of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0041] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions executed by the processor of the computer or other programmable data processing apparatus create means for performing the functions / operations specified in one or more blocks of the flowcharts or block diagrams, or both. These computer program instructions may also be stored in a computer readable medium capable of directing a computer, other programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, whereby the instructions stored in the computer readable medium include an article of manufacture including instructions implementing aspects of the functions / operations specified in one or more blocks of the flowcharts or block diagrams, or both.

[0042] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executing on the computer or other programmable apparatus provide a process for performing the functions / operations specified in one or more blocks of the flowcharts or block diagrams or both.

[0043] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts may represent a module, segment, or portion of code, including one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions shown in the blocks may occur in a different order than that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or these blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that executes the specified functions or operations, or executes a combination of dedicated hardware and computer instructions.

[0044] It should be noted that any of the methods described herein may include the additional step of providing a system including different software modules implemented on a computer readable storage medium, which may include, for example, any or all of the components detailed herein. The method steps may then be performed using the different software modules or sub-modules, or both, of the system, as described above, executed on the hardware processor 302. Furthermore, the computer program product may include a computer readable storage medium having code adapted to be implemented to perform at least one method step described herein, including providing a system with different software modules.

[0045] In each case, it will be understood that the components shown herein may be implemented in various forms of hardware, software, or combinations thereof, such as application specific integrated circuits (ASICS), functional circuits, a suitably programmed digital computer with associated memory, etc. Given the teachings of the present invention provided herein, one of ordinary skill in the art will be able to conceive other implementations of the components of the present invention.

[0046] Moreover, it is to be understood in advance that implementation of the teachings presented herein is not limited to any particular computing environment: rather, embodiments of the present invention may be implemented in conjunction with any type of computing environment now known or later developed.

[0047] For example, cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with a service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0048] Its features are as follows.

[0049] On-Demand Self-Service: Cloud consumers can automatically and unilaterally provision computing capacity, such as server time and network storage, as they need it, without the need for human interaction with the provider of the service.

[0050] Pervasive Network Access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (eg, cell phones, laptops, and PDAs).

[0051] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated on demand. Consumers are location independent in that they generally have no control or knowledge of the exact location of the resources provided to them, although they may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0052] Rapid Elasticity: Capabilities can be rapidly provisioned and rapidly scaled out, and rapidly released and rapidly scaled in, rapidly and elastically, sometimes automatically. To the consumer, the capabilities available for provisioning often appear unlimited, and can be purchased in any quantity at any time.

[0053] Metered Services: Cloud systems automatically control and optimize resource usage by using metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being utilized.

[0054] The service model is as follows:

[0055] Software as a Service (SaaS): The functionality provided to the consumer is the use of the provider's applications running on a cloud infrastructure. These applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or individual application capabilities, with the expected exception of limited user-specific application configuration settings.

[0056] Platform as a Service (PaaS): The functionality offered to the consumer is the deployment onto a cloud infrastructure of applications the consumer creates or acquires, written using programming languages ​​and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the configuration of the environment that hosts the applications.

[0057] Infrastructure as a Service (IaaS): The functionality provided to a consumer is to provision processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and possibly limited control over the selection of network components (e.g., host firewalls).

[0058] The deployment model is as follows:

[0059] Private Cloud: The cloud infrastructure is operated exclusively for an organization. It can be managed by that organization or a third party and can exist on-premise or off-premise.

[0060] Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by those organizations or by a third party and can reside on-premise or off-premise.

[0061] Public Cloud: The cloud infrastructure is available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0062] Hybrid Cloud: A cloud infrastructure is a blend of two or more clouds (private, community, or public) that remain unique entities but are tied together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0063] Cloud computing environments are service-oriented and focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0064] 4, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which a local computing device used by a cloud consumer, such as, for example, a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or a combination thereof, can communicate. The nodes 10 can communicate with each other. The nodes 10 can be physically or virtually grouped in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or a combination thereof, as described above (not shown). This enables the cloud computing environment 50 to provide Infrastructure as a Service, Platform as a Service, or Software as a Service, or a combination thereof, without the cloud consumer having to maintain resources on a local computing device. It will be understood that the types of computing devices 54A-N shown in FIG. 4 are intended to be merely exemplary, and that the computing node 10 and cloud computing environment 50 are capable of communicating with any type of computerized device (e.g., using a web browser) via any type of network or network-addressable connection or both.

[0065] Referring now to Figure 5, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 4) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 5 are intended to be merely illustrative, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0066] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, RISC (Reduced Instruction Set Computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0067] The virtualization layer 70 provides an abstraction layer that may provide the following examples of virtual entities: virtual servers 71, virtual storage 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75. In one example, the management layer 80 may provide the functionality described below: Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to perform tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and charging or billing for the consumption of these resources.

[0068] In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, and protection for data and other resources. A user portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources so that required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provides pre-allocation and procurement of cloud computing resources for forecasted future needs according to SLAs.

[0069] The workload tier 90 provides examples of functionality that can utilize a cloud computing environment. Examples of workloads and functionality that can be provided from this tier include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instructional delivery 93, data analytics processing 94, transaction processing 95, and collaborative neural networks with spatial confinement constraints 96 according to one or more of the present invention.

[0070] The terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting of the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, it will be understood that the terms "comprise" and / or "comprising" as used herein specify the presence of stated features, integers, steps, operations, elements or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components or groups, or combinations thereof.

[0071] At least one embodiment of the present invention can provide advantageous effects, such as, for example, productive use of partially labeled data that does not contain all the labels required by the entire task, incorporating domain knowledge to improve the speed and accuracy of training a network, and learning from unlabeled data.

[0072] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but they are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over the art found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for training a neural network by computer information processing, comprising the steps of: Dividing a processing task into a number of sub-tasks; training a plurality of independent neural networks, each of the plurality of sub-tasks corresponding to a respective one of the plurality of independent neural networks, the training including training the plurality of independent neural networks such that each of the plurality of independent neural networks detects at least two types of objects, the at least two types of objects having at least one spatial relationship to one another, the at least one spatial relationship including at least one containment relationship; determining, for a given one of the plurality of sub-tasks, at least one constraint loss that depends on outputs from at least one other of the plurality of sub-tasks based at least in part on constraint-based domain knowledge regarding the processing task; retraining at least a portion of the plurality of independent neural networks, the retraining including incorporating the at least one constraint loss into the at least one independent neural network to account for violations of the at least one spatial relationship between one or more outputs of at least one independent neural network of the plurality of independent neural networks and one or more outputs of another one or more independent neural networks of the neural networks, for a given input; Including, The method is performed by at least one computing device. method.

2. The method of claim 1 , wherein the sub-tasks correspond to different portions of a given object.

3. The method of claim 1 , wherein one of the plurality of sub-tasks corresponds to at least a portion of another of the plurality of sub-tasks.

4. The method of claim 1 , wherein the processing task includes determining a structure and location of one or more tables within a document.

5. 2. The method of claim 1, comprising back-propagating the at least one constraint loss throughout the at least one neural network by modifying weights assigned to each of one or more units in the at least one neural network toward values ​​that would have produced an output that satisfies a constraint.

6. The method of claim 1 , wherein the retraining comprises using the at least one constrained loss in a supervised setting.

7. The method of claim 1 , wherein the retraining comprises using the at least one constraint loss in an unsupervised setting.

8. A computer program comprising program instructions causing a computer to carry out a method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing the computer program of claim 8.

10. Memory, operatively coupled to the memory; Divide the processing task into multiple sub-tasks, training a plurality of independent neural networks, each of the plurality of sub-tasks corresponding to a respective one of the plurality of independent neural networks, the training including training the plurality of independent neural networks such that each of the plurality of independent neural networks detects at least two types of objects, the at least two types of objects having at least one spatial relationship to one another, the at least one spatial relationship including at least one containment relationship; determining, for a given one of the plurality of sub-tasks, at least one constraint loss that depends on outputs from at least one other of the plurality of sub-tasks based at least in part on constraint-based domain knowledge regarding the processing task; retraining at least a portion of the plurality of independent neural networks, the retraining including incorporating the at least one constraint loss into the at least one of the plurality of independent neural networks to account for violations of the at least one spatial relationship between one or more outputs of at least one of the plurality of independent neural networks and one or more outputs of one or more other of the neural networks, for a given input; At least one processor configured to A system including:

11. A method for training a neural network by computer information processing, comprising the steps of: Dividing a processing task into a number of sub-tasks; training a neural network having a plurality of branching branches, each of the plurality of sub-tasks corresponding to a respective one of the plurality of branching branches of the neural network, the training including training the neural network to detect at least two types of objects, the at least two types of objects having at least one spatial relationship to one another, the at least one spatial relationship including at least one containment relationship; determining, for a given one of the plurality of sub-tasks, at least one constraint loss that depends on outputs from at least one other of the plurality of sub-tasks based at least in part on constraint-based domain knowledge regarding the processing task; retraining at least a portion of the neural network, the retraining including incorporating the at least one constraint loss into the at least one branch branch considering that for a given input, there is a violation of the at least one spatial relationship between one or more outputs of at least one branch branch of the multiple branch branches of the neural network and one or more outputs of another one or more branch branches of the multiple branch branches of the neural network; Including, The method is performed by at least one computing device. method.

12. The method of claim 11 , wherein the sub-tasks correspond to different portions of a given object.

13. The method of claim 11 , wherein one of the plurality of sub-tasks corresponds to at least a portion of another of the plurality of sub-tasks.

14. 12. The method of claim 11, comprising back-propagating the at least one constraint loss across the at least one branching branch by modifying weights assigned to each of one or more units in the at least one branching branch toward values ​​that would have produced an output that satisfies a constraint.

15. A computer program comprising program instructions for causing a computer to carry out a method according to any one of claims 11 to 14.

16. A computer-readable storage medium storing the computer program of claim 15.

Images