Systems and methods for learning sensory medium associations without text labels

Abstract

Systems and methods for learning sensory medium associations without text labels. A computer-implemented method for learning sensory medium associations includes receiving a first type of non-text input and a second type of non-text input, encoding and decoding the first type of non-text input using a first autoencoder having a first convolutional neural network and encoding and decoding the second type of non-text input using a second autoencoder having a second convolutional neural network, bridging a first autoencoder representation associated with a first modality and a second autoencoder representation associated with a second modality with a deep neural network that learns a mapping between the first autoencoder representation and the second autoencoder representation, and generating a first type of non-text output and a second type of non-text output based on the first type of non-text input or the second type of non-text input under the first modality or the second modality based on the encoding, decoding, and bridging.

Description

TECHNICAL FIELD

[0001] Aspects of example implementations relate to methods, systems, and user experiences associated with learning sensory media associations (e.g., audio and / or visual) without using text labels. BACKGROUND

[0002] Prior art deep learning techniques require large amounts of text-labeled data. Text label data is created by human labeling for training models. The cost of performing text labeling in the prior art limits the use of deep learning techniques in many real-world situations.

[0003] For example, it is burdensome and expensive to create a custom product image dataset with millions of image labels using prior art deep learning techniques, sometimes to the extent that performing such a task is prohibitive. Furthermore, the detailed descriptions of images that are required to create videos with appropriate text labels by prior art deep learning techniques would also require a prohibitive cost in the form of human labeling in the form of tasks such as record review and typing.

[0004] Accordingly, there is a pressing need in prior art deep learning techniques to collect real-time data and create datasets without the costs and drawbacks associated with text labeling of the prior art. SUMMARY

[0005] According to aspects of example implementations, a computer-implemented method of learning sensory media associations includes receiving a first type of non-text input and a second type of non-text input; encoding and decoding the first type of non-text input using a first autoencoder having a first convolutional neural network and encoding and decoding the second type of non-text input using a second autoencoder having a second convolutional neural network; bridging a first autoencoder representation learned with the first modality and a second autoencoder representation learned with the second modality by a deep neural network that learns a mapping between the first autoencoder representation and the second autoencoder representation; and based on the encoding, decoding, and bridging, generating a first type of non-text output and a second type of non-text output based on the first type of non-text input under a first modality or the second type of non-text input under a second modality.

[0006] According to further aspects, the first type of non-text input is audio and the second type of non-text input is an image. According to other aspects, the audio is sensed by a microphone and the image is sensed by a camera.

[0007] According to other aspects, the first type of non-text input is one of audio, image, temperature, touch, and radiation and the second type of non-text input is another of audio, image, temperature, touch, and radiation.

[0008] According to other aspects, the first type of non-text input and the second type of non-text input are provided to the autonomous robot for training.

[0009] According to further aspects, no text labels are used, and receiving, encoding, decoding, bridging, and generating are language independent.

[0010] According to further aspects, a third type of non-text input is received, the third type of non-text input is encoded using a third autoencoder having a third convolutional neural network, the third autoencoder is bridged to the first and second autoencoders by a deep neural network that learns a mapping between a third type of representation associated with a third modality and the first and second types of representations, and the third type of non-text output is generated without retraining of the first, second, third autoencoders, the first, second, and third convolutional neural networks.

[0011] Example implementations can also include a non-transitory computer- readable medium having stored thereon instructions that, if executed by a processor, can cause the processor to perform steps of any of the methods described herein. BRIEF DESCRIPTION OF DRAWINGS

[0012] The patent or application file contains at least one drawing executed in color ink. Copies of this patent or patent application publication with color drawings will be provided by the Office upon request and payment of the necessary fee.

[0013] Figure 1 Example implementations of systems and methods are illustrated.

[0014] Figure 2 Results associated with example implementations are illustrated.

[0015] Figure 3 Results associated with example implementations are illustrated.

[0016] Figure 4 Results associated with example implementations are illustrated.

[0017] Figure 5 Results associated with example implementations are illustrated.

[0018] Figure 6 Results associated with example implementations are illustrated.

[0019] Figure 7 Results associated with example implementations are illustrated.

[0020] Figure 8 Results associated with example implementations are illustrated.

[0021] Figure 9An example process is shown in accordance with example implementations.

[0022] Figure 10 An example computing environment having an example computer device suitable for some example implementations is shown.

[0023] Figure 11 An example environment suitable for some example implementations is shown.

[0024] Figure 12 Example implementations associated with applications of robots are shown. DETAILED DESCRIPTION

[0025] The following detailed description provides further details of the drawings and example implementations of the present application. For clarity, reference numbers and descriptions of redundant elements between the figures are omitted. The terminology used throughout the description is provided by way of example and is not intended to be limiting.

[0026] The prior art is in need of tools that allow deep learning techniques to operate machine learning for sensory media without text labels. As noted above, the prior art approach involves the cost of obtaining text labeled data, which creates a bottleneck for many data demanding machine learning tasks. On the other hand, a human can learn cross-media associations without text labels (e.g., a child can learn how to name objects without knowing the open digital character, or from an alphanumeric perspective, an object can learn how to name objects in a language he or she does not know).

[0027] Aspects of example implementations involve cross-modal speech-visual associations without text labels. While the prior art approach can use text as a bridge to connect speech and visual data, example implementations involve machine learning using sensory media in a non-text manner (e.g., without a keyboard).

[0028] By eliminating text (e.g., keyboard notation), there can be various benefits and advantages. For example, and without limitation, machine learning techniques can be performed in a manner that more naturally and accurately mimics human behavior, and are not limited by the prior art limitations of keyboard notation (e.g., schedules, costs, etc.). As a result, the prior art problem of insufficient training data for machine learning tasks can also be alleviated; further, new training data areas can become available.

[0029] Further, according to example implementations, because there are no costs or related complexities associated with text labeling, ordinary consumers can be able to more easily train systems in ways that are currently not possible with prior art systems. For example, and without limitation, example implementations can be used to assist individuals with visual or hearing impairments, such that visual input can be provided as audio output for individuals with visual impairments, and audio input can be provided as visual output for individuals with hearing impairments.

[0030] According to example implementations, multiple deep convolutional autoencoders are provided. More specifically, one deep convolutional autoencoder is provided for a first non-textual domain (e.g., learning speech representations), and another deep convolutional autoencoder is provided for a second non-textual domain (e.g., learning image representations). Thus, hidden features can be extracted. The hidden spaces of these autoencoders represent compact embeddings of speech and images, respectively. Thus, two deep networks are trained to bridge the hidden spaces of the two autoencoders, which generates robust mappings of speech to images and images to speech. Thus, audio can be converted to images that can be visualized by a user. With these mappings, image input can activate corresponding speech output, and vice versa.

[0031] Example implementations associated with the inventive concept can be employed in a variety of situations. For example, and without limitation, systems can be used to assist the disabled; further, autonomous robot training can be performed, and machine learning algorithms and systems can be generated that can use large amounts of low-cost training data. Further, machine learning systems can be employed that are not limited by the prior art problems and drawbacks (e.g., cost, scheduling, etc.) associated with text labeling approaches.

[0032] In example implementations of the invention, a machine can be provided with sensors such as cameras and microphones that can continuously collect real-time data, similar to the way a human can sense the same information. Other sensors can be provided, such as a thermometer associated with temperature sensing, a pressure sensitive array associated with forming a pressure map to sense touch, a radiation sensor, or other sensors associated with sensed parameter information. The collected real-time data is used by the encoder and decoder architecture of example implementations of the invention. For example, a sensing device can obtain usable data from normal everyday activities, as well as from existing videos. Without the prior art limitations of labeling this data by human text labeling as in prior art approaches, example implementations can continuously sense and observe information of an environment, and learn from the environment.

[0033] Figure 1An example implementation of the architecture 100 is shown. More specifically, audio input 101 and image input 103 are provided, which can receive information from devices such as microphones and cameras, respectively. The example implementation includes an encoder and decoder architecture for each of the audio and image modules to learn audio and image representations, respectively. Through an encoding process 109, audio output 105 is generated, and through an encoding process 111, image output 107 is generated. Since the audio module uses audio signals as training inputs and outputs, no text labels are needed to train the deep network. Similarly, the image module uses images as inputs and outputs for the network, and no text labels are needed as well.

[0034] With the representations between each pair of encoder and decoder, one neural network is used to map audio representation 113 to image representation 115, and another neural network is used to map image representation 119 to audio representation 117. According to example implementations of the present invention with the above-described arrangements and learning parameters, the audio input can activate both the audio output as well as the image output. Conversely, the image input can activate both the image output as well as the audio output.

[0035] More specifically, according to example implementations, for each modality (as explained herein, Figure 1 Two modalities are shown in the center, but example implementations are not limited thereto, and additional modalities can be provided), the autoencoder includes respective encoder portions 121 and 123 that receive respective inputs 101 and 103 (in this case, audio and video modalities), respectively. After applying several layers of the encoder portions 121 and 123 to the input information, a first modality representation is generated as shown at 125, and a second modality representation is generated as shown at 127.

[0036] The first and second modality representations at 125 and 127 are then provided to a deep neural network to perform cross-modality bridging, such as a mapping from the first modality representation 113 to the second modality representation 115, or a mapping from the second modality representation 119 to the first modality representation 117. The sending and receiving of the representations are shown by the dashed lines extending from the representations 125 and 127.

[0037] Furthermore, as mentioned above, decoder portions 129 and 131 are provided to decode the respective first and second modality representations at 125 and 127, which include the results of the cross-modality bridging. After applying several layers of the decoder portions 129 and 131 to the first and second modality representations at 125 and 127, the outputs are generated at 105 and 107, respectively.

[0038] The example implementations described above can be used with different input-output combinations. For example, and not by way of limitation, when the architecture described above does not have information about the pairing between the audio input and the learned audio output, the example implementations can feed the input signal to both the input and the output of the audio module and can use an autoencoder learning process to learn the representation. When the pairing information between the audio input and the existing audio output is known, the example implementations can learn to associate the audio input and the existing audio output through the autoencoder. When both the audio output and the image output are available, the example implementations can train using both the output and the audio input. Conversely, similar methods using the example implementation architecture can be applied in a similar manner to train the image module.

[0039] The example implementations learn the relationship between the images and the audio clips. More specifically, the pairing information between the audio clips and the images is presented to the system associated with the example implementations. The pairing according to the example implementations is similar to the pairing that occurs when a person teaches another person to name objects. Thus, the example implementations provide a more natural learning method to the machine learning. With the pairing information provided by the machine teacher, the training Figure 1 The corresponding parameters in the network shown.

[0040] More specifically, according to one example implementation, the adversarial convolutional autoencoder is used for both the image and audio learning modules to save the low-level feature computation cost, and to reduce the number of training parameters, the audio input is converted to a 2D MFCC representation, which is fed to the convolutional autoencoder. This conversion results in an audio learning module that is very similar to the image learning module. The autoencoder includes seven layers for its encoder and decoder, respectively. However, the example implementations of the present invention are not limited thereto, and other numbers of layers can be substituted without departing from the scope of the present invention.

[0041] According to the example implementations, a 3x3 convolutional filter is used to process the data for each convolutional layer. The autoencoder compresses the input audio, which according to one example can have 16384 samples, 232 dimensional autoencoder intermediate layer, without loss of input fidelity. With this 32 dimensional representation of the input, the example implementations can reconstruct similar audio with the decoder without audible distortion.

[0042] With respect to the images, the 28x28 handwritten images are reshaped into two 784 dimensional vectors and fed to the image autoencoder. The image autoencoder has five fully connected layers to reduce the input to a 32 dimensional image representation. The 32 dimensional image representation can be used to reconstruct the input image with the trained decoder.

[0043] Figure 2Spectrogram and image 200 corresponding to different hidden node values on the grid located in the latent space when hidden nodes are used are shown. These show data clustering and the latent space. At 201, the audio learning module output is provided in the form of a spectrogram corresponding to different hidden node values. At 203, the image learning module output image corresponding to different hidden node values is provided. A two node latent space is provided for visualization, but this can result in loss of information and greater distortion of the output. To avoid these drawbacks and issues, and to keep the audio encoder output less distorted, the example implementation uses 32 bit nodes for both the audio learning module and the image learning module.

[0044] To learn the mapping between the 32 node audio representation layer and the 32 node image representation layer, two five layer fully connected networks of 512 nodes per layer are used to learn the mapping from audio to image and from image to audio, respectively.

[0045] The example implementation described above is applied to data in the following illustrative example. The NMIST handwritten digit dataset (with 60,000 training images and 10,000 test images) and the English spoken digit dataset from FSDD (with three speakers and 1500 recordings (50 per digit per speaker)) are used as training data for adjusting the network parameters.

[0046] Figure 3 Examples 300 of input audio spectrograms 301, 307 generated from audio input using the image decoder, corresponding audio learning module spectrogram outputs 303, 309 and corresponding output images 305, 311 are shown. When audio from different speakers is fed to the learning system, the image output has less variation on the digit output.

[0047] As Figure 4 shown at 400, typical handwritten images and speech activated images are provided using image input 401 and image output 403 as shown herein, the output images can be more recognizable than the input images. As Figure 4 shown, this is particularly evident for the digits 6, 7 and 8.

[0048] In addition, the 512 node latent space autoencoder is tested for both the images of the image module and the audio of the audio module using a GAN to learn the mapping from image to audio.

[0049] As Figure 5 shown at 500, the input 501 and output of the image learning module 503 and the corresponding audio spectrogram output 505 activated by the image input 501 are shown. Figure 5 The images shown show that due to the latent space expansion, the images of the image module can output images that are more similar to the image input.

[0050] Figure 6Test results 600 are shown for the coil-100 dataset, which includes input 601, autoencoder output 603, and speech output 605. Since the images in this dataset are large, a convolutional autoencoder is used to extract 512-dimensional features for representing the input images.

[0051] Further, using the abstract scene dataset, speech information is generated for 10,000 128x128 images. Using the learning architecture described above, the image representation layer and the audio representation layer are each scaled up to 1024 nodes. Similarly, the audio-to-image and image-to-audio mapping networks are scaled from 512 to 2048 to account for the increased data complexity.

[0052] Figure 7 Results for this example are shown at 700. More specifically, Figure 7 The first row shows real data 701, and the second row shows audio-generated images 703.

[0053] Figure 8 MFCC coefficients 800 are shown for three speech segments 801, 803, 805 generated using images. By having a witness listen to the image-activated speech segments, it is determined whether the speech segments are easily understood.

[0054] To enhance training quality, an example implementation can employ a trainer with an ID as a token. For the mode of showing an image and then generating speech, the token can be a random speaker or a specified speaker. On the other hand, for the mode of speaking and then generating an image, the result should be speaker-agnostic, so that the example implementation can operate according to one or more of the following options.

[0055] According to one example implementation, separate encoder-decoder models can be trained for both cases. In other words, one encoder-decoder model can be speaker-agnostic, in other words involving speech-to-image, while the other encoder-decoder model can use a token and involve image-to-speech.

[0056] According to another example implementation, a combined model can be trained, which uses a token and also has a token set ID for all speakers. This combined model will be trained twice on each tone. Alternatively, if there is a large amount of data, the tones can be randomly assigned to either a speaker token or an “everyone” token.

[0057] According to another example implementation, speaker IDs can be used. However, according to this example implementation, the speaker to which the system will pay attention can be limited to those with a speaker ID. This approach can be used in certain environments, such as at an airport, where officials can be trying to match an individual to a photo, and a more accurate and quick determination can be made where a dialect sensor and a speaker ID associated with the individual are present. Using this approach, clustering in the audio module can be performed in an easier and cleaner manner.

[0058] The example implementations described herein can have various implementations and applications. As described above, aspects of the example implementations can be used to build systems that can assist the disabled, particularly those that can provide visual or audio output that does not involve typing or entering information from a keyboard or mouse key, which can require fine motor skills. In addition, the example implementations can also be used in areas such as autonomous robot training, which requires robots to learn audio and visual environments in a manner similar to humans to be able to perform safely and efficiently in that environment. In addition, the example implementations can involve machine learning algorithms and / or systems that require large amounts of low-cost training data, and machine learning systems that aim to be free from the limitations of text labels (e.g., schedules, costs, etc.).

[0059] According to one example implementation, a language-independent device can be trained to help a person with a hearing impairment determine the conversational objects of other people around that person, or to use speech to inform a person with a visual impairment of the physical environment around him or her.

[0060] Since no text is used in the example implementations of the present invention, the training system is also language-independent and can be used across countries, cultures, and languages. Since the example implementations can include multiple sensors connected to a common network, users in the same area and speaking the same language are able to train the system in a common manner.

[0061] According to another example implementation related to autonomous robot training, the example approach is superior to sharing a latent space or a function-bounded latent space. More specifically, according to the example implementation, the decoupling of the latent space allows users to add more modalities to the machine at a later time without causing the new modalities to affect the old learned modalities. Instead, according to the example implementation, the new modalities will learn themselves and build more connections with the old modalities over time.

[0062] For example and without limitation, autonomous robots initially have sensors (e.g., cameras) relating to the visual modality and another sensor (e.g., microphones) relating to the audio modality. However, a user can wish to add additional sensors relating to other modalities (e.g., temperature, touch, radiation, or other parameters sensible in the environment). Those new modalities can be added to the example implementation in ways that are not possible in the prior art without affecting the already existing modalities (e.g., visual and audio). Moreover, the robots can allow for learning associated with environments that are difficult for human operation (e.g., deep sea, outer space, etc.).

[0063] According to one example implementation associated with the touch modality, a robot can be taught how to grasp an object such as a bottle or a glass. The robot can learn from its own training data associated with touch to determine whether to grasp the object with too little force or too much force. Since there is no notion of textual labeling, the robot can use its own output as the sensed input, or can learn from prior provided human training data.

[0064] Figure 9 An example process 900 according to an example implementation is shown. As explained herein, the example process 900 can be performed on one or more devices.

[0065] At 901, various types of non-textual input are received from sensing devices. For example and without limitation, audio input can be received from a microphone as one type of non-textual input, and image input can be received from a camera as another type of non-textual input. The example implementation is not limited to just two types of non-textual input, and other non-textual inputs can be included according to example implementations, such as temperature, touch, radiation, video, or other inputs that can be sensed.

[0066] At 903, auto-encoding and decoding is performed for each type of non-textual input for which input has been received. For example, auto-encoding and decoding can be performed using a convolutional neural network. Thus, audio input received from a microphone can be encoded by an auto-encoder, and image input received from a camera can be encoded by another auto-encoder. The output can be generated using a deep convolutional auto-encoder that learns representations for each respective type of non-textual input.

[0067] At 905, a deep network is used to bridge the latent spaces of the two deep convolutional autoencoders used at 903. More specifically, a deep neural network that learns a mapping between the first modality representation and the second modality representation is used to bridge the latent spaces between the first type of autoencoder representation and the second type of autoencoder representation. For example, and without limitation, a deep network is provided such that a mutual conversion can be performed between an audio type of input and an image type of output, or vice versa. When both audio and image outputs are available, an example implementation can train using both the audio and image outputs with the audio input; a similar approach can be taken for image inputs (when available). When paired information is not available, the autoencoder training can be performed using historical data.

[0068] At 907, based on the encoding, decoding, and bridging, for each type of non-text input, the non-text input in the first modality or the second modality generates an appropriate output comprising the first type of non-text output and the second type of non-text output. For example, an audio learning module output spectrogram or an output image corresponding to various hidden node values can be provided as output. Examples of inputs and outputs are shown in the above-referenced figures and are also described in the description of the above example implementations.

[0069] Figure 10 An example computing environment 1000 is shown having an example computing device 1005 suitable for use with some example implementations. The computing device 1005 in the computing environment 1000 can include one or more processing units, cores, or processors 1010, memory 1015 (e.g., RAM, ROM, etc.), internal storage 1020 (e.g., magnetic, optical, solid-state, and / or organic storage), and / or I / O interfaces 1025, any of which can be coupled on a communication mechanism or bus 1030 for communicating information or embedded in the computing device 1005.

[0070] The computing device 1005 can be communicatively coupled to an input / interface 1035 and an output device / interface 1040. Either or both of the input / interface 1035 and the output device / interface 1040 can be a wired or wireless interface and can be removable. The input / interface 1035 can include any device, component, sensor, or interface (physical or virtual) that can be used to provide input (e.g., buttons, touch screen interface, keyboard, pointing / cursor control, microphone, camera, Braille, motion sensor, optical reader, etc.).

[0071] Output device / interface 1040 can include a display, a television, a monitor, a printer, a speaker, Braille, etc. In some example implementations, input / interface 1035 (e.g., a user interface) and output device / interface 1040 can be embedded or physically coupled to computing device 1005. In other example implementations, other computing devices can function as or provide the functionality of input / interface 635 and output device / interface 1040 for computing device 1005.

[0072] Examples of computing device 1005 can include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles and other machines, devices carried by people and animals, etc.), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, server devices, other computers, kiosks, televisions, radios embedded with and / or coupled to one or more processors, etc.).

[0073] Computing device 1005 can be communicatively coupled (e.g., via I / O interface 1025) to external storage 1045 and network 1050 for communication with any number of networked components, devices, and systems, including one or more computing devices of the same or different configuration. Computing device 1005 or any connected computing device can function as, provide services of, or be referred to as a server, a client, a thin server, a general purpose machine, a special purpose machine, or another label. For example, and without limitation, network 1050 can include a blockchain network and / or a cloud.

[0074] I / O interface 1025 can include, but is not limited to, wired and / or wireless interfaces using any communication or I / O protocol or standard (e.g., Ethernet, 802.11 xs, universal system bus, WiMAX, modem, cellular network protocols, etc.) for communicating information to and / or from all connected components, devices, and networks in computing environment 1000. Network 1050 can be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0075] Computing device 1005 can use and / or communicate using computer-usable or computer-readable media, including transitory media and non-transitory media. Transitory media includes transmission media (e.g., metal wires, optical fibers, signals, carrier waves, etc.). Non-transitory media includes magnetic media (e.g., magnetic disks and magnetic tapes), optical media (e.g., CD ROMs, digital video disks, Blu-ray disks), solid state media (e.g., RAM, ROM, flash memory, solid state storage devices), and other non-volatile storage devices or media.

[0076] The computing device 1005 can be used in some example computing environments to implement techniques, methods, applications, processes, or computer-executable instructions. Computer-executable instructions can be retrieved from a transitory medium and stored on, and retrieved from, a non-transitory medium. The executable instructions can originate from one or more of any programming, scripting, and machine languages (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0077] The processor 1010 can execute under any operating system (OS) (not shown), in a native or virtual environment. One or more applications can be deployed, including a logic unit 1055, an application programming interface (API) unit 1060, an input unit 1065, an output unit 1070, a non-text input unit 1075, a non-text output unit 1080, an encoder / decoder and cross-media neural network unit 1085, and an inter-communication mechanism 1095 for the different units to communicate with each other, with the OS, and with other applications (not shown).

[0078] For example, the non-text input unit 1075, the non-text output unit 1080, and the encoder / decoder and cross-media neural network unit 1085 can implement one or more processes shown above for the above-described structures. The design, function, configuration, or implementation of the described units and elements can vary, not limited to the provided descriptions.

[0079] In some example implementations, when information or instructions are received through the API unit 1060, they can be communicated to one or more other units (e.g., the logic unit 1055, the input unit 1065, the non-text input unit 1075, the non-text output unit 1080, and the encoder / decoder and cross-media neural network unit 1085).

[0080] For example, the non-text input unit 1075 can receive and process inputs such as images and sounds, and generate corresponding image or sound outputs at the non-text output unit 1080 via processing by the encoder / decoder and cross-media neural network unit 1085 (e.g., using the above-described, in particular the above-described with respect to Figure 2 and Figure 5 disclosed).

[0081] In some cases, in some of the example implementations described above, logic unit 1055 can be configured to control the flow of information between control units and direct the services provided by API unit 1060, input unit 1065, non-text input unit 1075, non-text output unit 1080, and encoder / decoder and cross-media neural network unit 1085. For example, the flow of one or more processes or implementations can be controlled individually by logic unit 1055 or in combination with API unit 1060.

[0082] Figure 11 An example environment suitable for some example implementations is shown. Environment 1100 includes devices 1105-1145, and each device is communicatively connected to at least one other device via, for example, network 1160 (e.g., via wired and / or wireless connection). Some devices may be communicatively connected to one or more storage devices 1130 and 1145.

[0083] Examples of one or more devices 1105-1145 may be respectively Figure 10 The computing device 1005 described herein. Devices 1105-1145 may include, but are not limited to, a computer 1105 (e.g., a laptop computer) having a monitor and associated network camera as described above, a mobile device 1110 (e.g., a smartphone or tablet), a television 1115, a device associated with the vehicle 1120, a server computer 1125, computing devices 1135-1140, and storage devices 1130 and 1145.

[0084] In some implementations, devices 1105-1120 may be considered user devices associated with users of the enterprise. Devices 1125-1145 may be devices associated with service providers (e.g., used by external hosts to provide services as described above and with reference to the various figures, and / or to store data such as web pages, text, text portions, images, image portions, audio, audio clips, videos, video clips and / or related information).

[0085] Figure 12Example implementations associated with applications of robots are shown. More specifically, at 1200, a robot is represented. The robot can include sensors 1201 coupled by direct connection or wireless communication to provide input to the robot. Multiple sensors can be provided, each associated with one or more modalities. At 1203, a storage device is provided that includes instruction information (e.g., executable computer instructions) associated with example implementations of the present application, as well as data received from the sensors 1201. At 1205, a processor such as a microprocessor or CPU is provided that receives instructions and data from the storage device 1203 (which can be remote from the robot or located within the robot). Further, it is noted that the sensors 1201 can also provide data directly to the processor 1205, either remotely from the robot or within the robot.

[0086] The processor 1205 performs the various operations described in the above example implementations and generates output commands and data. The output commands and data can be provided to, for example, a player at 1207 that outputs information in one or more modalities, as well as a device at 1209 that performs an action (e.g., a motor or the like). Although Figure 12 Although the figure shows communication over a network, the elements shown therein can be connected directly to one another, e.g., using internal circuitry of the robot 1200, without departing from the scope of the present application.

[0087] The above example implementations can have various advantages and benefits over the prior art. For example, and without limitation, prior art machine learning methods explore style transfer within a single modality, but for cross-sensory media associations, the prior art only uses text labels as a branch. The example implementations take advantage of the advances and widespread adoption of IOT-type sensors (e.g., camera and microphone sensors) to provide novel ways of associating audiovisual sensory data without the need for text labels.

[0088] Further, although there are prior art methods that convert speech to text and use the text to retrieve images. However, speech-to-text requires a pre-defined speech recognition engine, whereas the above example implementations do not require a pre-existing speech engine in order to facilitate machine learning. The prior art methods that require a pre-existing speech engine also create difficulties for performing machine learning directly from sensory data.

[0089] Further, unlike prior art methods that use a common latent space for images and speech, example implementations involve using a mapping between two embeddings. More specifically, using a common latent space as in the prior art requires the system to replace a single shared latent space with respective separate latent spaces, which in turn significantly increases the modality dimension, in addition, a target function is introduced to force the two separate spaces to be close to each other. This prior art approach can also cause interference between different modalities. By using example implementations of the present invention that include a learning structure involving decoupled learning of each modality, and separately generating a non-linear modality link, the prior art problems and drawbacks associated with modality interference are avoided, while example implementations can continue to learn non-linear relationships between two modalities.

[0090] Further, unlike prior art methods that involve data from only one modality (e.g., text), example implementations build bridges between two different modalities (e.g., images and audio). Thus, example implementations are able to handle data that has asymmetric dimensions and structure across two modalities, which is not addressed by prior art solutions. Further, using a lookup table instead of a neural network approach is not an option compared to prior art lookup tables, because due to space and storage limitations on lookup tables, the ability to achieve the same functionality with a lookup table as with example implementations using CNN-based autoencoders as described above is not achievable if attempted would result in inefficient use of memory space.

[0091] While certain example implementations have been shown and described, these example implementations are provided so that the subject matter described herein can be conveyed to those familiar with the art. It will be understood that the subject matter described herein can be practiced with variations not specifically described, or with other or different elements or steps, without departing from the subject matter as defined in the claims and their equivalents. Those skilled in the art will understand that changes can be made to these example implementations without departing from the subject matter as defined in the claims and their equivalents.

Claims

1. A computer-implemented method of learning sensory modality associations, the method comprising the steps of: receiving a first type of non-textual input and a second type of non-textual input; encoding and decoding the first type of non-textual input using a first autoencoder having a first convolutional neural network and encoding and decoding the second type of non-textual input using a second autoencoder having a second convolutional neural network; bridging the first autoencoder representation and the second autoencoder representation by learning a first deep neural network mapping from a first autoencoder representation associated with a first modality to a second autoencoder representation associated with a second modality and learning a second deep neural network mapping from the second autoencoder representation associated with the second modality to the first autoencoder representation associated with the first modality, wherein the second deep neural network is different from the first deep neural network; and generating a first type of non-textual output and a second type of non-textual output based on the first type of non-textual input under the first modality or the second type of non-textual input under the second modality based on the encoding, decoding, and bridging.

2. The computer-implemented method of claim 1, wherein, The first type of non-textual input is audio and the second type of non-textual input is an image.

3. The computer-implemented method of claim 2, wherein, The audio is sensed by a microphone and the image is sensed by a camera.

4. The computer-implemented method of claim 1, wherein, The first type of non-textual input is one of audio, image, temperature, touch, and radiation and the second type of non-textual input is another of audio, image, temperature, touch, and radiation.

5. The computer-implemented method of claim 1, wherein, The first type of non-textual input and the second type of non-textual input are provided to an autonomous robot for training.

6. The computer-implemented method of claim 1, wherein, No textual labels are used and the receiving, encoding, decoding, bridging, and generating steps are language independent.

7. The computer-implemented method of claim 1, wherein, receiving a third type of non-textual input, encoding the third type of non-textual input using a third autoencoder having a third convolutional neural network, bridging the third autoencoder to the first autoencoder and the second autoencoder by learning the deep neural network mapping between the third type of representation associated with a third modality and the first type of representation and the second type of representation, and generating a third type of non-textual output without retraining the first autoencoder, the second autoencoder, the first convolutional neural network, and the second convolutional neural network.

8. A non-transitory computer-readable medium having executable instructions stored in storage and executed on a processor, the instructions comprising: receiving a first type of non-textual input and a second type of non-textual input; encoding and decoding the first type of non-textual input using a first autoencoder having a first convolutional neural network and encoding and decoding the second type of non-textual input using a second autoencoder having a second convolutional neural network; bridging the first autoencoder representation and the second autoencoder representation by learning a first deep neural network mapping from a first autoencoder representation associated with a first modality to a second autoencoder representation associated with a second modality and learning a second deep neural network mapping from the second autoencoder representation associated with the second modality to the first autoencoder representation associated with the first modality, wherein the second deep neural network is different from the first deep neural network; and based on the encoding, decoding, and bridging, generating a first type of non-textual output and a second type of non-textual output based on the first type of non-textual input under the first modality or the second type of non-textual input under the second modality.

9. The non-transitory computer-readable medium of claim 8, wherein, the first type of non-textual input is audio and the second type of non-textual input is an image.

10. The non-transitory computer-readable medium of claim 9, wherein, the audio is sensed by a microphone and the image is sensed by a camera.

11. The non-transitory computer-readable medium of claim 8, wherein, the first type of non-textual input is one of audio, image, temperature, touch, and radiation and the second type of non-textual input is another of audio, image, temperature, touch, and radiation.

12. The non-transitory computer-readable medium of claim 8, wherein, the first type of non-textual input and the second type of non-textual input are provided to an autonomous robot for training.

13. The non-transitory computer-readable medium of claim 8, wherein, no textual labels are used and the receiving, encoding, decoding, bridging, and generating are language independent.

14. The non-transitory computer-readable medium of claim 8, wherein, receiving a third type of non-textual input, encoding the third type of non-textual input using a third autoencoder having a third convolutional neural network, bridging the third autoencoder to the first autoencoder and the second autoencoder by the deep neural network learning a mapping between a third type of representation associated with a third modality and the first type of representation and the second type of representation, and generating a third type of non-textual output without retraining the first autoencoder, the second autoencoder, the first convolutional neural network, and the second convolutional neural network.

15. A computer-implemented system for learning sensory modality associations, the system comprising: a first type of sensor that receives a first type of non-textual input and a second type of sensor that receives a second type of non-textual input; a processor that receives the first type of non-text input and the second type of non-text input, encodes and decodes the first type of non-text input using a first autoencoder having a first convolutional neural network and encodes and decodes the second type of non-text input using a second autoencoder having a second convolutional neural network, and bridges the first autoencoder representation and the second autoencoder representation by learning a first deep neural network that maps from a first autoencoder representation associated with a first modality to a second autoencoder representation associated with a second modality and learning a second deep neural network that maps from the second autoencoder representation associated with the second modality to the first autoencoder representation associated with the first modality, wherein the second deep neural network is different from the first deep neural network; and an output device that, based on the encoding, decoding, and bridging, generates a first type of non-text output and a second type of non-text output based on the first type of non-text input under the first modality or the second type of non-text input under the second modality.

16. The computer-implemented system of claim 15, wherein, The first type of sensor is a microphone and the second type of sensor is a camera.

17. The computer-implemented system of claim 15, wherein, The first type of non-text input is one of audio, image, temperature, touch, and radiation and the second type of non-text input is another of audio, image, temperature, touch, and radiation.

18. The computer-implemented system of claim 15, wherein, The first type of non-text input and the second type of non-text input are provided to an autonomous robot for training.

19. The computer-implemented system of claim 15, wherein, No text labels are used and the receiving, encoding, decoding, bridging, and generating are language independent.

20. The computer-implemented system of claim 15, wherein, a third type of non-text input is received, the third type of non-text input is encoded using a third autoencoder having a third convolutional neural network, the third autoencoder is bridged to the first autoencoder and the second autoencoder by learning the deep neural network that maps between the third type of representation associated with a third modality and the first type of representation and the second type of representation, and a third type of non-text output is generated without retraining the first autoencoder, the second autoencoder, the first convolutional neural network, and the second convolutional neural network.

Images