Neural network system for abstract reasoning

Abstract

A computer-implemented method, system, and computer program product for solving a cognitive task involving learning abstract properties. The method can include accessing a dataset characterizing abstract properties. The accessed dataset can then be input into a first neural network to generate first embeddings. Pairs of the generated first embeddings can be formed, the pairs of first embeddings corresponding to pairs of the dataset. Data corresponding to the formed pairs can then be input into a second neural network, which can be executed to generate second embeddings. The latter can capture relational properties of the pairs of the dataset. Subsequently, a third neural network can be executed based on the generated second embeddings to obtain output values. One or more abstract properties of the dataset are learned based on the obtained output values in order to solve the cognitive task.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of computer-implemented methods and systems for solving cognitive tasks that require learning abstract properties. In particular, the present disclosure relates to methods that rely on coupled neural networks to generate embeddings that capture relational properties of pairs of input datasets (e.g., frames), where such datasets characterize the abstract properties to be learned. In some embodiments, a mutual information-based regularizer can be advantageously used to regularize the neural networks. BACKGROUND

[0002] The development of the EDVAC system in 1948 is often cited as the beginning of the computer era. Since then, computer systems have evolved from extremely complex devices into extremely complex devices. Today's computer systems typically include a combination of complex hardware and software components, applications, operating systems, processors, buses, memory, input / output devices, and the like. As advances in semiconductor processing and computer architecture push the performance of the computer higher and higher, even more advanced computer software has evolved to make use of these capabilities, resulting in today's computer systems being much more powerful than just a few years ago.

[0003] One application of these new capabilities is machine learning. Machine learning relies heavily on artificial neural networks (ANN), which generally refer to computational models inspired by biological neural networks in the human or animal brain. Such systems typically learn tasks by example in a step-wise and autonomous manner; they have been successfully applied to speech recognition, text processing, and computer vision, among many other tasks.

[0004] Many types of ANNs are known, starting with feed-forward neural networks (FFNN), such as multilayer perceptrons, deep neural networks (DNN), and convolutional neural networks (CNN). ANNs are typically implemented in software; the vast majority of implementations of ANNs involve software implementations. However, ANNs can also be implemented in hardware, for example as resistive processing units or optical neuromorphic systems.

[0005] The visual oddity task generally refers to a set of tasks designed to test basic concepts of geometry, such as points, lines, parallelism, and symmetry. It was first introduced as part of a neuroscience experiment to test knowledge of the principles of geometric concepts. Each task contains six frames, five of which include the geometric concept being tested. One frame violates the geometric concept, referred to as the oddity. The goal of the task is to classify which of the six frames is the oddity. SUMMARY

[0006] According to a first aspect, a computer-implemented method of solving a cognitive task requiring learning of an abstract property. The method first comprises accessing a dataset of frames (e.g., image frames) that are representative of the abstract property. Then, the accessed dataset is input into a first neural network (e.g., a CNN), the first neural network is executed accordingly to generate first embeddings. Next, pairs of the generated first embeddings are formed, the pairs of first embeddings corresponding to pairs of the dataset. Then, data corresponding to the formed pairs (e.g., data obtained by concatenating the pairs of first embeddings) is input into a second neural network, the second neural network is executed to generate second embeddings. The latter capture relational properties of the pairs of the dataset. Subsequently, a third neural network is executed based on the generated second embeddings to obtain output values that can be generally interpreted as distances or similarities between the accessed dataset. Each of the second neural network and the third neural network can be a fully connected neural network. Finally, one or more abstract properties of the dataset are learned based on the obtained output values in order to solve the cognitive task.

[0007] In some embodiments, the method further comprises regularizing at least one of the first neural network and the second neural network using a mutual information based regularizer, e.g., by applying a regularization term determined by minimizing a mutual information based function, e.g., according to a variational approximation of the information bottleneck function.

[0008] In some embodiments, the accessed dataset respectively correspond to frames representative of the abstract property, and the cognitive task is a visual oddity task, whereby one of the accessed frames represents an oddity.

[0009] In some embodiments, each of the first neural network, the second neural network and the third neural network is executed based on parameters learned during one or more previous phases while solving one or more previous different cognitive tasks.

[0010] According to another aspect, there is provided a computerized system for solving a cognitive task, which includes learning abstract properties in line with the present method. In operation, the system is adapted to run an input unit, a neural network module, and an output unit. The input unit is adapted to access a dataset characterizing abstract properties. The neural network module is connected to the input unit. The module includes a first neural network, a structure operation unit, a second neural network, and a third neural network. The first neural network is configured to generate first embeddings upon execution based on the dataset accessed by the input unit. The structure operation unit is configured to form pairs of the first embeddings generated by the first neural network, the formed pairs corresponding to pairs of the dataset. The second neural network is configured to generate second embeddings capturing relational properties of the pairs of the dataset upon execution based on the data corresponding to the pairs formed by the structure operation unit in operation. The third neural network is configured to produce output values (e.g., interpretable as distances or similarities between the accessed dataset) upon execution based on the second embeddings generated by the second neural network. Finally, the output unit is connected to the neural network module and is configured to learn one or more abstract properties of the dataset based on the output values obtained from the third neural network to solve the cognitive task.

[0011] According to another aspect, a computer program product (again for solving a cognitive task including learning abstract properties). The computer program product includes a computer readable storage medium having program instructions embodied therewith, wherein the program instructions are executable by one or more processors to cause implementation of operations in accordance with the present method.

[0012] The computerized system, method, and computer program product embodying the present disclosure will now be described by way of non-limiting examples and with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are used to further illustrate various embodiments and explain various principles and advantages in accordance with the present disclosure, in which like reference numerals refer to like or similar elements throughout the several views, and which drawings are incorporated in and form a part of the specification and are meant to explain the principles of and the practical application of the present embodiments, in which:

[0014] Figure 1 is a flowchart showing high-level components of a computerized system in accordance with various embodiments and operations of a method of operating the system to solve a cognitive task as in various embodiments;

[0015] Figures 2A to 2D shows a set of different frames (i.e., images) that can be used as input to the neural network module of the system depicted in Figure 1 to solve a visual oddity task as in embodiments;

[0016] Figure 3is a diagram schematically illustrating the input to and output from a neural network of a system as depicted in Figure 1

[0017] Figure 4 A general computerized system, suitable for implementing one or more method operations involved in embodiments of the present disclosure, is schematically represented.

[0018] The accompanying drawings show simplified representations of computerized systems or parts thereof as involved in embodiments. Similar or functionally similar elements in the figures are assigned the same reference numerals unless stated otherwise. DETAILED DESCRIPTION

[0019] The present disclosure describes solutions to visual oddity tasks, or more generally cognitive tasks requiring analogical formation.

[0020] ANNs can generally comprise a set of connected units or nodes, which are thus referred to as artificial neurons in comparison to biological neurons in animal brains. Similar to synapses, signals can be transmitted along connections between artificial neurons, also referred to as edges. In summary, an artificial neuron receiving a signal can process this signal and can then signal connected neurons.

[0021] Connection weights, also referred to as synaptic weights, can be associated with connections, and such weights can be adjusted as learning proceeds. Each neuron can have several inputs, and a connection weight can be attributed to each input (the weight of that particular connection). Connection weights can be learned during a training phase by a training algorithm and are thereby updated. The learning process can be iterative: data instances can be presented to the network, typically one at a time, and weights associated with input values can be adjusted at each time step.

[0022] Reference is made to Figures 1 to 3 , which describes an aspect of the present disclosure, which relates to a computer-implemented method of solving a cognitive task requiring learning of abstract properties. Note that the present method and variants thereof are collectively referred to as “the present method”. All references to Sij refer to method operations as described in Figure 1 and 3 , while numeric references refer to components of system 1 and digital objects consumed or produced by it.

[0023] As shown in Figure 1 , the method can first comprise accessing S10 a dataset 10, which may, for example, comprise frames (e.g. images). More generally, the dataset can comprise data files or data characterizing abstract properties to be learned in order to solve a given cognitive task. However, in the following, for the sake of illustration, it is assumed that dataset 10 is frames such as those depicted in Figures 2A to 2D .​

[0024] The accessed frames 10 can be input S20 to a first neural network 21, which can be a CNN. This network can be executed based on the input frames 10 to generate S30 a first embedding 31, which can be the output from the first neural network 21, which embedding can generally be a feature vector, e.g. a one-dimensional (ID) array of numerical values.

[0025] Next, pairs 32 of the first embeddings 31 can be formed S40. The pairs 32 of embeddings can also correspond to pairs of frames 10, which task can be performed by the structural operation unit 21a, which forms part of the neural network modules 21-23 implementing a pipeline of three neural networks, as described below. The pairs 32 can be simply formed S40 by concatenating the data corresponding to each pair of embeddings 31, e.g. the data corresponding to the formed pairs can comprise concatenated values which ultimately form a single vector 32 for each pair of embeddings 31. Note that, Figure 3 The pairs 32 of embeddings 31 described in the foregoing can thus comprise each one in a single array.

[0026] The data corresponding to the formed pairs 32 can then be input S40 to a second neural network 22, which can be a fully connected network. As described, the input data can simply comprise concatenated data of the pairs of embeddings 31, such that a one-dimensional array of data can be input S40 to the second network 22, which can then be executed based on this data to generate S50 second embeddings 33, 34, which can be the output from the second neural network 22, which capture relational properties of the pairs of frames 10.

[0027] Next, based on the second embeddings 33, 34, a third neural network 23 can be executed. The direct output 33 from the network 22 can undergo some mathematical operations before being fed S50 to the third network 23, resulting in transformed data 34. In some embodiments, all outputs 33 from the second neural network 22 corresponding to the same frame can be summed before being input S50 to the third neural network 23 for execution of the third neural network 23. The third network 23 can also be a fully connected network. The execution of the third network 23 can provide S60 an output value 35, which can generally be interpreted as a distance or similarity between the accessed frames 10.

[0028] Next, one or more abstract properties of the frames 10 can be learned based on the output value 35 obtained at operation S60, to solve a cognitive task. The cognitive task can for example be a visual anomaly task, whereby one of the frames 10 accessed at operation S10 represents an anomaly when compared to the other frames, see for example Figures 2A to 2DIn this case, the abstract properties of the frames of learning (based on the output values 35 obtained at operation S60) make it possible to solve the cognitive task by distinguishing one from the others in the input frames based on the properties determined for each frame.

[0029] In addition to visual oddity tasks and other visual tasks, embodiments of the present disclosure can allow the solution of certain cognitive tasks that can be characterized by the input dataset 10 to be effectively solved by the relational network formed by the second network 22 and the third neural network 23, the idea behind these embodiments being to first compute the embeddings 31 from the input dataset 10 and then form pairs 32 of the embeddings to obtain data 32 (e.g., concatenated vectors) that can be fed into the second neural network 22. In this way, second embeddings 33, 34 can be generated that can capture relational properties of pairs of frames 10. Using a network pipeline 22, 23 designed to handle such relational properties, data 34 obtained from the second embeddings 33 can then be fed into the third network 23 to obtain outputs 35 that can be simply metered to learn the abstract properties desired and thereby solve the cognitive task, as discussed in more detail later.

[0030] In some embodiments, the method can also include regularizing at least one (possibly each) of the first neural network 21 and the second neural network 22, network regularization can be performed using a mutual information based regularizer. Mutual information generally refers to a measure of the mutual dependence of two random variables and can be used, in essence, to measure the relationship between such variables. Mutual information can be thought of as a reduction in uncertainty of a random variable given the available knowledge about the other variable. It can be measured in bits, also known as Shannon. In the present context, mutual information can include the information between the hidden layers and the inputs and outputs of the neural network.

[0031] In some embodiments, regularization can be achieved by employing a mutual information estimate as an additional term in the loss function characteristic of the network. The regularization term can be added to the total loss function as an additional loss contribution along with the cross-entropy. Tests performed by the authors of the present disclosure have shown that such regularization can stabilize test accuracy and can significantly reduce variance. A comparison of the accuracy achieved on individual tasks indicates that the network pipeline 21-23 can achieve overall performance on visual oddity tasks that is substantially above human accuracy when augmented by a mutual information based regularizer.

[0032] In some embodiments, the regularization term can be determined by minimizing a mutual information based function. For example, the regularization term can be determined according to a variational approximation to the known Information Bottleneck (IB) method.

[0033] The aim of the IB method is to find a maximally compressed representation of the input random variable X, which can be obtained as a function of a relevant random variable Y, such that it retains as much information as possible about the relevant random variable Y. A known alternative mutual information estimation method in neural networks is a variational approximation of the IB method, known as the Variational Information Bottleneck (VIB), which can be exploited in the current context. The VIB method can be applied to multiple layers of a convolutional neural network for network compression: an approximation of the mutual information between successive layers can be added as a regularizer as part of the objective function.

[0034] For example, VIB-based regularizers can be added to the convolutional layers of the convolutional neural network and the second neural network 22 (e.g. the four fully connected layers that make it up). The VIB-based regularizers can improve and stabilize the accuracy obtained by the network structure, in particular for visually singular tasks.

[0035] As mentioned before, the first network can be a CNN, followed by fully connected layers until the final output. One feature and advantage of a CNN is its benefit for sensory image data. In some embodiments, the first network 21 is a CNN, which can comprise four, five or more layers. The second network 22 and / or the third network 23 can in turn be fully connected networks. The network 22 can for example comprise four layers, while the third neural network can comprise three layers. The four layers of the second network can for example each comprise 512 neurons, while the three layers of the third network can for example comprise 256, 256 and 1 neurons, respectively. More generally, successive networks can be designed to accommodate successively compressed data (e.g. from the input dataset 10 to the first embedding 31 to the second embedding 33, 34 to the final output 35).

[0036] The second and third networks 22, 23 can be implemented as relational network modules, which form part of the overall network module 21-23, as described later with reference to another aspect of the disclosure.

[0037] In a first type of embodiment, each initial image frame 10 can be input S20 into a four-layer CNN 21. Each convolutional layer can for example have 32 filters of kernel size 5x5. The convolutional output S30 for each image from the convolutional neural network can then be concatenated S40 per channel, thereby flattening it into a vector (i.e. a one-dimensional array). The flattened output can be input S40 into a four-layer fully connected network 22.

[0038] In some of such first type of embodiments, batch normalization can be used for all convolutional layers. Moreover, dropout regularization can be applied after all convolutional layers (e.g., with a dropout rate of 0.3) that are not preceded by a pooling layer and after the first fully connected layer of the second network 22 (e.g., with a dropout rate of 0.5).

[0039] As a whole, the neural network modules 21-23 implementing the three networks 21-23 can be configured as a relational network for solving the visual oddity task. Such network can be advantageously designed as an extension of the known Wild Relation Network (WReN), which was introduced to solve the Raven’s Progressive Matrices (RPM). However, the WReN model directly computes pairwise relations of the RPM panels, i.e., it determines pairwise relations between the context panel and the selection panel. For cognitive tasks such as the visual oddity task, there is no context panel or selection panel, since all frames can potentially be classified as odd.

[0040] In a second type of embodiments, each frame 10 of the visual oddity task can be input S20 into a 5-layer CNN 21, which can output a respective frame embedding 31. As shown in Figure 3 , the task may, for example, comprise identifying the non-quadrilateral from a given set of six frames, see also Figure 2A . For example, for each frame k e [1, 2,..., 6], the CNN 21 can output a frame embedding y k , which is denoted in Figure 3 by reference numeral 31, and then, for each frame embedding y k , a pair 32 can be generated S40 from an ordered concatenation with the other five frame embeddings, as described in Figure 3 using different patterned shapes 31, 32. As an illustrative example, for the first frame (frame 1), the required pairs are {yl, y2}, {yl, y3},..., and {yl, y6}. Thus, a total of 30 pairs 32 can be generated. Then, each pair 32 can be fed S40 to a second network 22. In other words, Figure 3 the architecture assumes that each pair 32 can be used as an argument of a function g θ parameterized by the neural network 22, and this is done for each frame. The output 33 of g θ corresponding to a given frame can be summed up, generating an array of data 34; the same type of computation can be performed for each frame. Then, the vector 34 of summed outputs corresponding to each frame 10 can be fed S50 to a third neural network 23 (e.g., which is used as an argument of a second function f parameterized by the network 23) to compute a final score 35, and then the oddity is identified based on all the final scores obtained.Figure 3 are shown.

[0041] In embodiments, learning of abstract properties can simply be done by applying a softmax function across all output values 35 obtained to determine the probability that each frame is singular, as further assumed in where singular corresponds to the maximum value 0.970 in this illustrative example. Figure 3

[0042] Other possible visual singularity tasks can for example include distinguishing non-convex shapes from convex shapes Figure 2B ), identifying pairs of shapes that cannot be obtained by rotation and other pairs of shapes that can be obtained by rotation Figure 2C ), or distinguishing non-equilateral triangles from equilateral triangles Figure 2D . Each singular frame is bounded by a dashed line in the datasets 10A-10D shown. Figures 2A-2D

[0043] The networks 21, 22, 23 can be computed for training or inference purposes and can be trained before performing inference. Network parameters learned for a given task can be transferred to additional tasks. In particular, in the present approach, each of the first, second and third neural networks 21, 22, 23 can be executed based on network parameters learned when executing the same network to solve one or more previous different cognitive tasks. Transfer learning is investigated in particular for visual singularity tasks, where each task provides a limited amount of data, for example, by initially training on 44 out of 45 tasks, the average test accuracy of a model trained on a small amount of samples of a previously unseen task is significantly increased. For example, if each task provides only 50 training samples, the accuracy of the transfer learning model is increased from 45.3% to 73.8%.

[0044] The above described embodiments have been described briefly with reference to the accompanying drawings, and can provide a variety of variations. Several combinations of the above described features can be considered.

[0045] ​​In these embodiments, the neural network architecture can be leveraged to solve cognitive tasks, which can additionally benefit from network regularization by employing the mutual information estimate as an additional term in the loss function characteristic of the network, especially for large training epochs. This regularization can provide benefits by stabilizing the test accuracy and significantly reducing its variance. The neural network structure of some embodiments can be further augmented by elements mediated by the estimate of mutual information for the solution of abstract reasoning challenges. Supervised learning techniques can be used to solve visual oddity tasks and achieve an average task accuracy of 96.5%. Through regularization based on the mutual information estimate, the accuracy can be further improved to 97.1%. Finally, the proposed model can benefit from and / or implement transfer learning.

[0046] Next, according to another aspect, embodiments of the present disclosure can be embodied as a computerized system 1 generally designed for solving cognitive tasks that can include learning abstract properties. Operational aspects of this system 1 have been implicitly described with reference to the present method. The system 1 is only briefly described below.

[0047] In line with some embodiments, the system 1 can be equipped with a processor(s) and a memory, the latter including computerized methods allowing the system 1 to form, in operation, an input unit 20, a neural network module 21-23 and an output unit 24, referred to Figure 1 and 4 in the present method.

[0048] The input unit 20 can be designed to access frames 10 characterizing abstract properties. The neural network module 21-23 can be connected to the input unit 20. As previously mentioned, the neural network module 21-23 can include a first neural network 21, a structure operation unit 21a, a second neural network 22 and a third neural network 23.

[0049] In more detail, the first neural network 21 can be configured to generate, upon execution based on the frames 10 accessed by the input unit 20, first embeddings 31. The structure operation unit 21a can be designed to form pairs 32 of the first embeddings 31 generated by the first neural network 21, the formed pairs 32 can correspond to pairs of the frames 10, the second neural network 22 can be configured to generate, upon execution based on data corresponding to the pairs 32 formed by the structure operation unit 21a, second embeddings 33, 34 that can capture relational properties of the pairs 32 of the frames 10. The third neural network 23 can be designed to produce, upon execution based on the second embeddings 33, 34 generated by the second neural network 22, output values 35 that can be interpreted as distances or similarities between the accessed frames 10. Finally, the output unit 24 can be connected to the neural network module 21-23 and can be configured to learn one or more abstract properties of the frames 10 based on the output values 35 that can be obtained from the third neural network 23 to solve cognitive tasks.

[0050] In an embodiment, system 1 may also be operable to form the first neural network 21 as a CNN, while the second neural network 22 and / or the third neural network 23 may be formed as a fully connected neural network. As previously described, neural network modules 21-23 may also be configured to use a mutual information-based regularizer, for example by applying a regularization term determined according to the VIB method, to regularize one or each of the first neural network 21 and the second neural network 22.

[0051] Computerized systems can be suitably designed to implement embodiments of the present disclosure as described herein. In this regard, it will be understood that some embodiments described herein are largely non-interactive and automated. However, in some embodiments, the methods described herein can also be implemented in interactive, partially interactive, or non-interactive systems. Furthermore, the methods described herein can be implemented in software (e.g., firmware), general-purpose hardware, special-purpose hardware (e.g., graphics processing unit), or a combination thereof. In some embodiments, the methods described herein can be implemented in software as an executable program, which is then executed by a suitable digital processing device. More generally, embodiments of the present disclosure can be implemented using a general-purpose digital computer, such as a personal computer, workstation, etc.

[0052] For example, Figure 4 The system 1 depicted schematically represents a computerized unit 101, such as a general-purpose computer. In some embodiments, in terms of hardware architecture, such as Figure 4 As shown, unit 101 may include processor 105, memory 110 coupled to memory controller 115, and one or more input and / or output (I / O) devices 145, 150, 155 (or peripherals) communicatively coupled via local input / output controller 135. Input / output controller 135 may be, but is not limited to, one or more buses or other wired or wireless connections. Input / output controller 135 may also have additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers, omitted for simplicity, to enable communication. Furthermore, the local interface may include address, control, and / or data connections to enable proper communication between the aforementioned components.

[0053] In some embodiments, processor 105 may be a hardware device for executing software, particularly software stored in memory 110. Processor 105 may be any custom or commercially available processor, central processing unit (CPU), auxiliary processor among several processors associated with computer 101, semiconductor-based microprocessor (in the form of a microchip or chipset), or any device typically used for executing software instructions.

[0054] In some embodiments, the memory 110 can include any one or a combination of volatile memory elements (e.g., random access memory) and nonvolatile memory elements. In addition, the memory 110 can include computer-readable storage media to store data files. Note that the memory 110 can have a distributed architecture, where various components are situated remotely from one another, but can be accessed by the processor 105.

[0055] The software in the memory 110 can include one or more separate programs, each of which comprises an ordered listing of executable program instructions for implementing logical functions. The software in the memory 110 includes a suitable operating system (OS) 111 in Figure 4 In the example, the software in the memory 110 includes some or all of the methods described herein according to some embodiments, and a suitable operating system (OS) 111. The OS 111 can substantially control the execution of other computer programs, and can provide scheduling, input-output control, file and data management, memory management, and communication control and related services.

[0056] The methods described herein can be in the form of a source program, executable program (object code), script, or any other entity comprising a set of instructions to be executed. When in the form of source program, the program can be translated via a compiler, assembler, interpreter, or the like, which can or can not be included within the memory 110. Furthermore, the methods can be written as an object oriented programming language, which has classes of data and methods, or a procedure programming language, which has routines, subroutines, and / or functions.

[0057] It is possible that the conventional keyboard 150 and mouse 155 can be coupled to the input / output controller 135. Other I / O devices 145-155 can include other hardware devices, such as a camera, microphone, printer, etc.

[0058] Additionally, the I / O devices 145-155 can further include devices that communicate both inputs and outputs. The system 100 can further include a display controller 125 coupled to a display 130. In some embodiments, the system 100 can further include a network interface or transceiver 160 for coupling to a network (not shown).

[0059] In some embodiments, the network transmits and receives data between the unit 101 and external systems. The network can be implemented in a wireless manner, for example, using wireless protocols and technologies such as WiFi, WiMax, etc. The network can be a fixed wireless network, a wireless local area network (LAN), a wireless wide area network (WAN), a personal area network (PAN), a virtual private network (VPN), an intranet, or other appropriate network system, and includes equipment for receiving and transmitting signals.

[0060] The network can also be an IP-based network for communication between the unit 101 and any external servers, clients, etc. via a broadband connection. In some embodiments, the network can be a managed IP network managed by a service provider. Further, the network can be a packet-switched network such as a LAN, WAN, the Internet, etc.

[0061] If the unit 101 is a personal computer (PC), workstation, smart device, etc., the software in the memory 110 can also include a basic input output system (BIOS). The BIOS can be stored in a read only memory (ROM) structure so that the BIOS can be executed when the computer 101 is activated.

[0062] When the unit 101 is in operation, the processor 105 can be configured to execute program instructions stored in the memory 110 to communicate data to and from the memory 110, and generally to control operations of the computer 101 pursuant to the software and OS 111. The methods described herein and the OS 111 can be read, in whole or in part, by the processor 105, usually buffered within the processor 105, then executed. When implementing the methods described herein in software, the methods can be stored on any computer readable medium, such as the storage device 120, for use by or in connection with any computer related system or method.

[0063] Aspects of the present disclosure can also be implemented as a computer program product. Generally, a computer program product can include a computer readable storage medium having program instructions embodied therewith, wherein the program instructions can be executed by one or more processors to cause a computer system 1 to implement operations as previously described, e.g., accessing a frame 10, executing a first neural network 21 to generate a first embedding 31 based on such input frame, forming a pair 32 of the resulting embeddings 31, inputting the respective data into a second neural network 22 and executing the latter to generate second embeddings 33, 34, which can capture relational properties of the pair 32 of the frame 10, finally, a third neural network 23 can be computed based on the second embeddings 33, 34 to obtain an output value 35 based on which one or more abstract properties of the frame 10 can be learned to solve a cognitive task.

[0064] Accordingly, aspects of the disclosure can include a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure. The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a solid state drive (SSD), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire,

[0065] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for execution by the computing device. Computer readable program instructions for carrying out aspects of the present disclosure can be

[0066] Program instructions and data stored on computer readable storage media can be transmitted in a transient signal, such as the electrical signals, electromagnetic signals, or digital signals contained in a network or other communications medium. Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for execution by the computing device. Computer readable program instructions for carrying out aspects of the present disclosure can be

[0067] Computer readable program instructions for carrying out operations of the present disclosure can be in assembly code, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0068] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0069] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0070] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0071] The flow and block diagrams in the drawings show the architectural, functional, and operational aspects of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow or block diagrams can represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession can in fact be executed substantially concurrently or in the reverse order, depending on the functionality involved. Also, it will be noted that each block of the block and / or flow diagram illustration, and combinations of blocks in the block and / or flow diagram illustration, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or combinations of hardware and software.

[0072] While the present disclosure has been described with reference to a limited number of embodiments, variations and drawings, those skilled in the art will understand that various changes can be made and equivalents can be substituted for elements without departing from the scope of the present disclosure. In particular, features recited in given embodiments, variations, or shown in the drawings (of similar devices or similar methods) can be combined or substituted for other features in another embodiment, variation, or drawing without departing from the scope of the present disclosure. Accordingly, various combinations of the features described with respect to any of the above-described embodiments or variations can be made, which remain within the scope of the appended claims. Furthermore, numerous minor modifications can be made to adapt a particular situation or material to the teachings of the present disclosure without departing from the scope thereof. Therefore, the present disclosure is not limited to the particular embodiments disclosed, but includes all embodiments falling within the scope of the appended claims. Moreover, many other variations are contemplated beyond those specifically mentioned above.

Claims

1. A computer-implemented method for solving a visual anomaly task comprising learning abstract attributes in image frames, the method comprising using a processor to: access a dataset from an input unit, the dataset comprising a plurality of image frames characterizing the abstract attributes; input the accessed dataset into a first neural network of a neural network module, the first neural network being a convolutional neural network for generating first embeddings based on execution of the dataset accessed by the input unit; form pairs of the generated first embeddings, wherein the formed pairs correspond to pairs of the image frames and generate a vector representation of a pair by concatenating data corresponding to the first embeddings of each pair of the pair; input data corresponding to the formed pairs into a second neural network of the neural network module, the second neural network being a fully connected neural network for capturing relational attributes of the pairs of the image frames; execute the second neural network to generate second embeddings capturing the relational attributes of the pairs of the image frames; execute a third neural network of the neural network module based on the second embeddings to obtain output values, the third neural network being a fully connected neural network for outputting a probability value that each image frame is an anomalous frame; and learn one or more abstract attributes of the plurality of image frames based on the obtained output values to identify anomalous frames in the image frames to solve the visual anomaly task.

2. The method of claim 1, further comprising regularizing at least one of the first neural network and the second neural network using a mutual information based regularizer.

3. The method of claim 2, wherein, Using the mutual information based regularizer comprises applying a regularization term determined by minimizing a mutual information based function.

4. The method of claim 3, wherein, The applied regularization term is determined from a variational approximation to an information bottleneck function.

5. The method of claim 1, wherein, The convolutional neural network comprises at least five layers.

6. The method of claim 1, wherein, The second neural network comprises four layers.

7. The method of claim 1, wherein, The third neural network comprises three layers.

8. The method of claim 1, wherein: forming the pairs of the first embeddings comprises concatenating the pairs of first embeddings; and the data corresponding to the formed pairs comprises concatenated values that form a single vector for each pair of the pairs of first embeddings.

9. The method of claim 1, wherein, All outputs from the second neural network corresponding to the same dataset in the dataset are summed prior to being input to the third neural network for execution of the third neural network.

10. The method of claim 1, wherein, Learning the one or more abstract attributes comprises applying a softmax function across the obtained output values to determine a probability that each frame is the anomaly.

11. The method of claim 1, wherein, Each of the first neural network, the second neural network, and the third neural network are executed based on neural network parameters learned while executing the first neural network, the second neural network, and the third neural network to solve one or more previous different cognitive tasks.

12. A computerized system for solving a visual anomaly task comprising learning abstract attributes in image frames, the system operable to form: an input unit adapted to access a dataset characterizing the abstract attributes, the dataset comprising a plurality of image frames; a neural network module connected to the input unit, wherein the neural network module comprises a first neural network, a structure operation unit, a second neural network, and a third neural network, wherein: the first neural network is configured to generate a first embedding upon execution based on a dataset accessed by the input unit, the first neural network being a convolutional neural network, the first neural network generating a first embedding based on execution of the dataset accessed by the input unit; the structure operation unit is configured to form pairs of the first embedding generated by the first neural network, the formed pairs corresponding to pairs of the image frames; the second neural network is configured to generate a second embedding capturing relational properties of the pairs of the image frames upon execution based on data corresponding to the pairs formed by the structure operation unit, the second neural network being a fully connected neural network; and the third neural network is configured to produce an output value upon execution based on the second embedding generated by the second neural network, the third neural network being a fully connected neural network for outputting a probability value that each image frame is a singular frame, and an output unit connected to the neural network module and configured to learn one or more abstract properties of the plurality of image frames based on the output value obtained from the third neural network to solve the visual singularity task.

13. The system of claim 12, wherein, the neural network module is further configured to regularize at least one of the first neural network and the second neural network using a mutual information based regularizer.

14. The system of claim 13, wherein, the neural network module is further configured to apply a regularization term determined from a variational approximation to an information bottleneck function when regularizing the at least one of the first neural network and the second neural network.

15. A computer program product for solving a visual singularity task comprising learning abstract properties in image frames, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by one or more processors to cause: accessing a dataset characterizing the abstract properties, the dataset comprising a plurality of image frames; inputting the accessed dataset into a first neural network and executing the first neural network to generate a first embedding, the first neural network being a convolutional neural network, the first neural network generating a first embedding based on execution of the accessed dataset; forming pairs of the generated first embedding, the formed pairs corresponding to pairs of the plurality of image frames; inputting data corresponding to the formed pairs into a second neural network and executing the second neural network to generate a second embedding capturing relational properties of the pairs of the plurality of image frames, the second neural network being a fully connected neural network; executing a third neural network based on the second embedding to obtain an output value, the third neural network being a fully connected neural network for outputting a probability value that each image frame is a singular frame; and learning one or more abstract properties of the plurality of image frames based on the obtained output value to solve the visual singularity task.

Images