Conversation resolution for enhanced context switching

Abstract

A computer receives multimedia data, the multimedia data including a plurality of frames. The computer converts the multimedia data into signal waves having a plurality of frequencies and a plurality of amplitudes. The computer determines a frame from the plurality of frames having a pronoun. The computer identifies a topic for the frame. The computer searches for a frame in a media repository having a highest correlation coefficient with the topic of the frame, the frame from the media repository includes a bag of objects, and resolves anaphora disambiguation by replacing the pronoun with an object from the bag of objects.

Description

Technical Field

[0001] The present invention generally relates to the field of computing, and more particularly to natural language processing.

Background Art

[0002] Natural language processing (NLP) is a field of computational linguistics related to the interaction between computers and human natural language, such as computer science, artificial intelligence, and programming computers for processing multimedia files incorporating large natural language corpora. Typically, the artificial intelligence component of NLP incorporates one or more neural networks trained to recognize or process natural language.

[0003] A neural network is a computational model in computer science based on a collection of neural units. Each neural unit is an artificial neuron that can be connected to other neural units to create a neural network. The neural network can then be trained to find solutions to problems where conventional computer programs, such as text embedding or word embedding in NLP, fail.

Summary of the Invention

[0004] According to one embodiment, a method, computer system, and computer program product for anaphora disambiguation are provided. The present invention may include a computer that receives multimedia data, which includes a plurality of frames. The computer converts the multimedia data into a signal wave having a plurality of frequencies and a plurality of amplitudes. The computer determines a frame from a plurality of frames having a pronoun. The computer identifies the topic of the frame. The computer searches for a frame in a media repository having the highest correlation coefficient with the topic of the frame, the frame from the media repository includes a bag of objects, and the computer resolves the anaphora disambiguation by replacing the pronoun with an object from the bag of objects.

[0005] These and other objects, features, and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention, which should be read in conjunction with the accompanying drawings. The various features of the drawings are not necessarily drawn to scale as the illustration is for clarity in enabling one skilled in the art to understand the invention in conjunction with the detailed description. The drawings include the following:

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Embodiments for Carrying Out the Invention

[0007] Although detailed embodiments of the structure and method of the claims are disclosed herein, it can be understood that the disclosed embodiments are merely exemplary of the structure and method of the claims that can be implemented in various forms. However, the present invention can be implemented in many different forms and should not be construed as being limited to the exemplary embodiments described herein. In the description, well-known features and technical details may be omitted in order to avoid unnecessarily obscuring the presented embodiments.

[0008] As described above, NLP is a field of computational linguistics related to the interaction between computers and human natural language, such as computer science, artificial intelligence, and programming computers for processing multimedia files incorporating large natural language corpora.

[0009] Employees perform multitasking throughout the workday to maintain many different roles necessary to support the business. For example, many software developers work in various languages on various projects. Software developers may also be involved in social media, marketing, invention, project management, and recruitment. Many information sources are consumed to accomplish tasks. At the same time, employees watch videos and listen to podcasts to help solve tasks. Many learning styles are in an indirect form and occur in the background while a person is working on a task. When a person performs both task switching and media consumption, the person confuses ambiguous pronouns with previous contexts or irrelevant information sources.

[0010] Anaphora is a natural language phenomenon when using one word (usually a pronoun) that refers to or replaces another word previously used in a sentence to avoid repetition. For example, in the sentence "Susan dropped the plate and it shattered with a loud noise", the pronoun "it" refers to the plate. Anaphora can also refer to something that complements rather than the antecedent. For example, in the sentence "The children who ate ice cream were few, and instead they threw it away around the room", the anaphora "they" refers to the children who did not eat ice cream rather than the children who ate ice cream.

[0011] Resolving anaphoric ambiguity not only aids in the NLP processing of multimedia content using natural language but may also be beneficial for consumers with cognitive impairments in consuming multimedia content, such as while watching videos or listening to podcasts. Therefore, it can be advantageous to implement a system that resolves anaphora in natural language, especially by converting part of the natural language into signals and determining similar signals.

[0012] According to one embodiment, an anaphora resolution process can be used to extract natural language data from a multimedia file, convert it into multiple sine waves, then compare it with a repository of previously resolved anaphora waves, and based on identifying similarities, resolve the anaphora using the resolved anaphora with the highest similarity waveform pattern.

[0013] The present invention may be a system, method, or computer program product, or a combination thereof, at any possible level of integration of technical details. The computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to execute aspects of the present invention.

[0014] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy (R) disks, punch cards, or mechanically encoded devices such as raised structures within grooves having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed to be a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted via a wire.

[0015] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to respective computing / processing devices, or downloaded from an external computer or an external storage device via a network such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within each respective computing / processing device.

[0016] The computer-readable program instructions for carrying out the operation of the present invention can be in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or object-oriented programming languages such as Smalltalk(R), C++, and the like, as well as procedural programming languages such as the "C" programming language or similar programming languages, and can be either source code or object code. The computer-readable program instructions may be executed as a stand-alone software package, entirely on the user's computer, partly on the user's computer, partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit in order to implement aspects of the present invention.

[0017] Aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0018] These computer readable program instructions may be provided to the processor of a computer, other programmable data processing apparatus, or other device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, other programmable data processing apparatus, or other device to function in a particular manner, such that the medium storing the instructions comprises an article of manufacture including instructions which implement the function / act specified in one or more blocks of the flowchart and / or block diagram.

[0019] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowchart and / or block diagram by causing a series of operational steps to be performed on the computer, other programmable apparatus, or other device.

[0020] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially simultaneously, or the blocks may be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams or flowchart diagrams, or combinations of blocks in the block diagrams or flowchart diagrams, can be implemented by a special-purpose hardware-based system that performs the specified functions or operations, or a combination of special-purpose hardware and computer instructions.

[0021] The exemplary embodiments described below provide a system, method, and program product for performing anaphora resolution in multimedia data by converting a chunk of data into a plurality of sine waves each having an amplitude and a frequency, comparing the sine waves based on their amplitudes and frequencies with waves associated with repository data, and resolving ambiguity by inserting anaphora resolution.

[0022] Referring to FIG. 1, an exemplary networked computer environment 100 according to at least one embodiment is depicted. The networked computer environment 100 may include client computing devices 102, a server 112, and a corresponding repository 122 interconnected via a communication network 114. According to at least one implementation, the networked computer environment 100 may include a plurality of client computing devices 102 and a server 112, and only one of each of them is shown for simplicity of illustration.

[0023] The communication network 114 may include various types of communication networks such as a wide area network (WAN), a local area network (LAN), a telecommunications network, a wireless network, a public switched network, or a satellite network, or a combination thereof. The communication network 114 may include connections such as wires, wireless communication links, or fiber optic cables. It will be understood that FIG. 1 provides only an illustration of one implementation and does not imply any limitations regarding the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.

[0024] Client computing device 102 may include a processor 104 and a data storage device 106 that host and execute software program 108 and disambiguation program 110A, and are enabled to communicate with server 112 via communication network 114. Client computing device 102 may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device that can execute a program and access a network. As will be discussed with reference to FIG. 3, client computing device 102 may include internal components 302a and external components 304a, respectively.

[0025] Server computer 112 may be a laptop computer, a netbook computer, a personal computer (PC), a desktop computer, or any programmable electronic device, or any network of programmable electronic devices, that hosts and executes disambiguation program 110B and database 116, and is enabled to communicate with client computing device 102 via communication network 114 according to an embodiment of the present invention. As will be discussed with reference to FIG. 3, server computer 112 may include internal components 302b and external components 304b, respectively. Server 112 may also operate in a cloud computing service model such as software as a service (SaaS), platform as a service (PaaS), or infrastructure as a service (IaaS). Server 112 may also be deployed in a cloud computing deployment model such as a private cloud, a community cloud, a public cloud, or a hybrid cloud.

[0026] The topic domain 118 may be a database that stores a large number of topic domains, along with previously resolved references, stored as a plurality of objects or bag of words. The topic domain can be extracted from an inactive media repository 122 by a word embedding algorithm or a trained neural network that can extract topics from, for example, natural language text using NLP methods.

[0027] The reference repository 122 can be a database, any programmable electronic device, or any network of a database or a programmable electronic device, or both, that can host and store a plurality of multimedia data such as video streams, audio streams, text, audio files, and video files. In another embodiment, the reference repository 122 can be dynamically determined using the search function of a web browser.

[0028] According to this embodiment, the reference ambiguity resolution programs 110A, 110B can be programs that analyze natural language, determine one or more reference sentences, convert the reference sentences into signals, and perform pronoun ambiguity resolution by comparing the signals with previously resolved media or inactive media using waveform similarity. The reference ambiguity resolution method will be described in more detail below with respect to FIG. 2.

[0029] Next, referring to FIG. 2, an operation flowchart showing a reference ambiguity resolution process 200 is depicted according to at least one embodiment. At 202, the reference ambiguity resolution programs 110A, 110B receive a multimedia object. The multimedia object can be any type of file such as audio, video, or text, or a data stream having natural language that is convertible to text and presented in any format. According to an exemplary embodiment, the reference ambiguity resolution programs 110A, 110B can receive the multimedia object from the client computing device 102.

[0030] Next, at 204, the reference ambiguity resolution programs 110A, 110B convert the multimedia object into a time-varying signal (i.e., a wave). According to an exemplary embodiment, the reference ambiguity resolution programs 110A, 110B convert the multimedia object into a signal wave by extracting audio data, or, if text is incorporated in the multimedia object, convert the text to audio by using an amplitude auto-decoder neural network that converts the text to amplitude values, and each value is assigned a time frame associated with the time at which the value was generated. In another embodiment, the reference ambiguity resolution programs 110A, 110B can apply a trained neural network that uses speech-to-text to convert the multimedia object to text and then uses an auto-decoder neural network to convert the text to a plurality of time-varying amplitudes. Additionally, the reference ambiguity resolution programs 110A, 110B can use a Fourier transform to convert the generated signal into a frequency spectrum and convert the signal into a sum of infinite sine waves for future comparison (see steps 210 and 212 below).

[0031] According to an exemplary embodiment, the ambiguity resolution programs 110A, 110B can convert a signal into a sum of infinite sine waves using the following equation

Number

[0032] The ambiguity resolution programs 110A, 110B can determine the frequency and phase of the generated signal by dividing the signal into time chunks (frames) using a spectrogram approach. According to an exemplary embodiment, the ambiguity resolution programs 110A, 110B can generate a spectrogram by means of a related fast Fourier transform such as a short-time Fourier transform (STFT) algorithm for continuous time. Using the determined frequency and phase, the signal can be plotted using the following equation

Number

[0033] The look-back window w is given by the following equation

Number

Number

[0034] Next, at 206, the anaphoric ambiguity resolution programs 110A, 110B divide the multimedia data and signals into frames. According to an exemplary embodiment, the anaphoric ambiguity resolution programs 110A, 110B can separate the multimedia objects and related signals into frames over a duration, with one or more frames containing anaphora. The duration of each frame may be determined based on the look-back time value m determined in the previous step.

[0035] Next, at 208, the echo ambiguity resolution programs 110A and 110B encode the signals into amplitude values using a deep neural network (DNN) and a word-to-vector method. According to an exemplary embodiment, the echo ambiguity resolution programs 110A and 110B may label the main objects in each frame using a DNN, and it is possible to encode each frame by applying a long-short term memory (LSTM) method. The labels can be associated with one or more topics within the topic domain 118. For each object, a vector describing the object, including speed, acceleration, and label, may be generated. If the multimedia object is video content, the hue color may be added to the vector as one of the parameters. If the DNN cannot determine the label for an object, the echo ambiguity resolution programs 110A and 110B may use a word-to-vector mapping (i.e., word embedding) method to determine similar words for the label. In a further embodiment, the amplitude of each of the encoded frames may be averaged.

[0036] Next, at 210, the echo ambiguity resolution programs 110A and 110B perform semantic encoding. According to an exemplary embodiment, if the position of echo ambiguity resolution is not identified by searching the topic domain 118 having the determined label, the echo ambiguity resolution programs 110A and 110B may perform semantic encoding by using sine wave signal decomposition to identify the relationship between an object and a plurality of objects in the inactive media repository 122. According to an exemplary embodiment, the echo ambiguity resolution programs 110A and 110B can decompose the relationship between a pair of objects using a discrete Fourier transform (DFT).

[0037] Next, at 212, the reference ambiguity resolution programs 110A, 110B identify the relevance of the content using the correlation coefficient. According to an exemplary embodiment, the reference ambiguity resolution programs 110A, 110B can use spectrograms such as a time-versus-amplitude spectrogram and a frequency-versus-amplitude spectrogram to compare the signal of the current object with a plurality of objects in the inactive media repository 122. According to an exemplary embodiment, the reference ambiguity resolution programs 110A, 110B may apply a fractional DFT to find the main waveform using the equations F n [f]=F[F n-1 [f]], and F n =(F -1 ) n , where F[f] is the continuous Fourier transform of the function f (i.e., the signal generated from each object), n is a non-negative integer, and F 0 [f]=f. According to an exemplary embodiment, the reference ambiguity resolution programs 110A, 110B can identify the correlation between the main waveform of one or more objects in the inactive media repository 122 and the identified object that may contain a reference within a frame from the multimedia data 120. According to an exemplary embodiment, the reference ambiguity resolution programs 110A, 110B can determine that the highest correlation coefficient between the main wave in the inactive media repository 122 and the identified object is associated with reference ambiguity resolution.

[0038] Next, at 214, the anaphora resolution programs 110A and 110B extract a bag of objects based on the highest correlation coefficient. According to an exemplary embodiment, the anaphora resolution programs 110A and 110B can identify all pronouns within the multimedia data 120 and identify the bag of objects, or bag of words, within the inactive media repository 122 associated with the highest correlation coefficient. According to an exemplary embodiment, the anaphora resolution programs 110A and 110B use a speech-to-text DNN to convert each frame to text, identify all pronouns based on searching the text, and using all available pronouns, can identify frames having pronouns that require disambiguation.

[0039] Then, at 216, the anaphora resolution programs 110A and 110B resolve the anaphora based on the bag of objects. According to an exemplary embodiment, the anaphora resolution programs 110A and 110B can replace words from the bag of words having the highest correlation coefficient in the identified pronouns. For example, the anaphora resolution programs 110A and 110B can replace a pronoun in the text with one of the words within the bag of words and incorporate the text into the multimedia data 120 such as the captions within the video. In another embodiment, the anaphora resolution programs 110A and 110B can replace the corresponding frame from the multimedia data 120 having the pronoun with a frame from the inactive media repository 122 having the highest correlation coefficient.

[0040] It will be understood that FIG. 2 provides only an illustration of one implementation and does not imply any limitation as to the ways in which different embodiments may be implemented. Many modifications to the depicted environment can be made based on design and implementation requirements.

[0041] FIG. 3 is a block diagram 300 of the internal and external components of the client computing device 102 and the server 112 depicted in FIG. 1, according to an embodiment of the present invention. It should be understood that FIG. 3 provides only an illustration of one implementation and does not imply any limitation regarding the environments in which different embodiments can be realized. Many modifications to the depicted environments can be made based on design and implementation requirements.

[0042] The data processing systems 302, 304 represent any electronic device capable of executing machine-readable program instructions. The data processing systems 302, 304 represent a smart phone, a computer system, a PDA, or other electronic devices. Examples of computing systems, environments, or configurations, or combinations thereof, represented by the data processing systems 302, 304 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held devices or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments including any of the above systems or devices.

[0043] Client computing device 102 and server 112 may each include a respective set of internal components 302a, b and external components 304a, b as shown in FIG. 3. Each set of internal components 302 includes one or more processors 320, one or more computer-readable RAMs 322, and one or more computer-readable ROMs 324 on one or more buses 326, as well as one or more operating systems 328 and one or more computer-readable tangible storage devices 330. One or more operating systems 328, software programs 108 and correspondence ambiguity resolution programs 110A in the client computing device 102, and the correspondence ambiguity resolution program 110B in the server 112 are each stored in one or more of the respective computer-readable tangible storage devices 330 for execution by one or more of the respective processors 320 via one or more of the respective RAMs 322 (usually including cache memory). In the embodiment shown in FIG. 3, each of the computer-readable tangible storage devices 330 is a magnetic disk storage device of an internal hard drive. Alternatively, each of the computer-readable tangible storage devices 330 is a semiconductor storage device such as a ROM 324, an EPROM, a flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0044] Each set of internal components 302a, b also includes an R / W drive or interface 332 for reading and writing with one or more portable computer-readable tangible memory devices 338, such as CD-ROMs, DVDs, memory sticks, magnetic tapes, magnetic disks, optical disks, or semiconductor memory devices. Software programs, such as cognitive screen protection programs 110A, 110B, can be stored in one or more of the respective portable computer-readable tangible memory devices 338, read out via the respective R / W drive or interface 332, and loaded onto the respective hard drive 330.

[0045] Each set of internal components 302a, b also includes a network adapter or interface 336, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, or a 3G or 4G wireless interface card or other wired or wireless communication link. Software programs 108 and disambiguation programs 110A within the client computing device 102, and disambiguation programs 110B within the server 112, can be downloaded to the client computing device 102 and the server 112 from an external computer via a network (e.g., the Internet, a local area network, or other wide area network) and the respective network adapter or interface 336. From the network adapter or interface 336, the software program 108 and disambiguation program 110A in the client computing device 102 and the disambiguation program 110B in the server 112 are loaded onto the respective hard drive 330. The network may include copper wire, optical fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers or combinations thereof.

[0046] Each of the sets of external components 304a, b can include a computer display monitor 344, a keyboard 342, and a computer mouse 334. The external components 304a, b can also include a touch screen, a virtual keyboard, a touch pad, a pointing device, and other human interface devices. Each of the sets of internal components 302a, b also includes a device driver 340 that interfaces with the computer display monitor 344, the keyboard 342, and the computer mouse 334. The device driver 340, the R / W drive or interface 332, and the network adapter or interface 336 comprise hardware and software (stored in the storage device 330 or ROM 324, or both).

[0047] This disclosure includes a detailed description of cloud computing, but it should be understood in advance that the implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in combination with any other type of computing environment, whether currently known or later developed.

[0048] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (such as networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0049] The characteristics are as follows: On-demand self-service: Cloud consumers can automatically and unilaterally provision computing capabilities such as server time and network storage as needed, without the need for human interaction with the service provider. Broad network access: The functions are available via the network and are accessed through standard mechanisms that facilitate use on heterogeneous thin-client platforms or thick-client platforms (e.g., mobile phones, laptops, and PDAs). Resource pooling: The provider's computing resources are pooled to provide services of different physical and virtual resources that are dynamically assigned and re-assigned to multiple consumers as needed, using a multi-tenant model. Consumers generally do not have control over or knowledge of the exact location of the resources provided, but there is a sense of location independence in that they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center). Rapid elasticity: The functions can be provisioned quickly and elastically, and in some cases automatically, to scale out rapidly and release quickly to scale in. To the consumer, the functions available for provisioning often appear limitless and can be purchased in any quantity at any time. Measured service: The cloud system automatically controls and optimizes resource use by leveraging metering capabilities at an abstraction level appropriate for the type of service (e.g., storage, processing, bandwidth, and active user accounts). Transparency is provided to both the provider and consumer of the service being utilized, as resource usage can be monitored, controlled, and reported.

[0050] The service model is as follows: Software as a Service (SaaS): The function provided as a service is to use the provider's applications running on the cloud infrastructure. These applications can be accessed from various client devices through a thin client interface such as a web browser (e.g., web-based email). Except for the possibility of limited user-specific application configuration settings, the consumer does not manage or control the underlying cloud infrastructure, including the network, server, operating system, storage, or individual application functions. Platform as a Service (PaaS): The function provided as a service is to deploy the applications created or obtained by the consumer, which are created using the programming languages and tools supported by the provider, to the cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure such as the network, server, operating system, or storage, but can control the deployed applications and, in some cases, the application hosting environment configuration. Infrastructure as a Service (IaaS): The function provided as a service is to provision processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, including the operating system and applications. The consumer does not manage or control the underlying cloud infrastructure, but can control the operating system, storage, deployed applications, and, in some cases, perform limited control of selected network components (e.g., host firewall).

[0051] The deployment model is as follows: Private cloud: The cloud infrastructure is operated exclusively for an organization, which may be managed by the organization or a third party and may exist on-premises or off-premises. Community cloud: The cloud infrastructure is shared by multiple organizations to support a specific community that shares concerns (such as mission, security requirements, policies, and compliance considerations). It may be managed by an organization or a third party and may exist on-premises or off-premises. Public cloud: The cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services. Hybrid cloud: The cloud infrastructure remains a distinct entity but is a composite of two or more clouds (private, community, or public) tied together by standardized technologies or proprietary technologies (such as cloud bursting for load balancing between clouds) that enable data and application portability.

[0052] The cloud computing environment is service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability. At the center of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0053] Referring now to FIG. 4, an exemplary cloud computing environment 50 is depicted. As shown, cloud computing environment 50 includes one or more cloud computing nodes 100 with which local computing devices, such as, for example, a mobile information terminal (PDA) or cellular phone 54A, desktop computer 54B, laptop computer 54C, or automotive computer system 54N, or a combination thereof, used by cloud consumers can communicate therewith. The nodes 100 can communicate with one another. They may be physically or virtually grouped in one or more networks, such as private, community, public, or hybrid clouds as described above in this specification, or a combination thereof (not shown). Thus, cloud computing environment 50 can provide infrastructure, platforms, software, or a combination thereof, as a service such that cloud consumers do not need to maintain resources on local computing devices. Computing devices 54A-N of the type shown in FIG. 4 are for illustrative purposes only, and it is understood that cloud computing nodes 100 and cloud computing environment 50 can communicate with any type of computerized device (e.g., using a web browser) via any type of network or network addressable connection, or both.

[0054] Referring now to FIG. 5, a set of functional abstraction layers 500 provided by cloud computing environment 50 is shown. It should be understood in advance that the components, layers, and functions shown in FIG. 5 are for illustrative purposes only and that embodiments of the invention are not limited thereto. As shown in the figure, the following layers, and corresponding functions are provided.

[0055] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61; RISC (Reduced Instruction Set Computer) architecture-based server 62; server 63; blade server 64; memory device 65; and network and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0056] The virtualization layer 70 provides an abstraction layer, from which the following examples of virtual entities may be provided: virtual server 71; virtual storage 72; virtual network 73 including virtual private network; virtual applications and operating systems 74; and virtual client 75.

[0057] In one example, the management layer 80 may provide the functions described below. Resource provisioning 81 provides for the dynamic procurement of computing resources and other resources utilized to execute tasks within a cloud computing environment. Metering and pricing 82 provides for cost tracking of resources when used within a cloud computing environment and for billing or invoicing for consumption of these resources. In one example, these resources may include application software licenses. Security provides for the protection of data and other resources in addition to identity verification for cloud consumers and tasks. The user portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides for cloud computing resource allocation and management to meet required service levels. Service level agreement (SLA) planning and fulfillment 85 provides for the advance arrangement and procurement of cloud computing resources for which future requirements are anticipated according to the SLA.

[0058] The workload layer 90 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this layer include mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis processing 94; transaction processing 95; and anaphora resolution 96. Anaphora resolution 96 is related to identifying pronouns within multimedia data, converting a pronoun frame to a wave signal, searching an inactive media repository for one or more frames having a signal wave with the highest correlation coefficient to the signal wave of a frame having a pronoun, and replacing the pronoun with a topic from the frame having the highest correlation coefficient.

[0059] The descriptions of various embodiments of the present invention are presented for illustrative purposes, but are not intended to be exhaustive or to limit the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, the practical application, or a technical improvement to the technology found in the marketplace, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A processor-implemented method for resolving anaphora ambiguity, comprising: receiving multimedia data, wherein the multimedia data includes a plurality of frames; converting the multimedia data into a signal wave, wherein the signal wave is converted into a plurality of sine waves having a plurality of frequencies and a plurality of amplitudes using a direct Fourier transform; identifying a frame from the plurality of frames having a pronoun; identifying the topic of the frame using a deep neural network; searching for a frame in a media repository having the highest correlation coefficient with the frame, wherein the frame from the media repository includes a bag of objects; resolving the anaphora ambiguity by replacing the pronoun with an object from the bag of objects; A processor-implemented method comprising the above steps.

2. The converting of the multimedia data into the signal wave comprises: converting the multimedia data into the plurality of amplitudes using an auto-decoder neural network; generating the signal wave from the plurality of amplitudes based on a time frame of each of the plurality of amplitudes; The method according to claim 1, comprising the above steps.

3. The method according to claim 1, wherein each of the plurality of frames has a duration, and the duration is determined based on a short-time Fourier transform of the signal wave.

4. The method according to claim 1, further comprising separating the signal wave into frames using a spectrogram approach. The method according to claim 1, comprising the above step.

5. The method according to claim 4, wherein the highest correlation coefficient is based on the spectrogram approach.

6. The method according to claim 1, further comprising identifying a label of the frame using a deep neural network. The method according to claim 1, comprising the above step.

7. The method according to claim 6, further comprising generating a vector describing the object, wherein the vector includes velocity, acceleration, and the label. The method according to claim 6, comprising the above step.

8. A computer system for resolving anaphora ambiguity, comprising: One or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more tangible storage media for execution by at least one of the one or more processors via at least one of the one or more memories, the computer system comprising receiving multimedia data, the multimedia data including a plurality of frames, said receiving converting the multimedia data into a signal wave, the signal wave being converted into a plurality of sine waves having a plurality of frequencies and a plurality of amplitudes using a direct Fourier transform, said converting identifying a frame from the plurality of frames having a pronoun identifying the topic of the frame using a deep neural network searching for a frame in a media repository having the highest correlation coefficient with the frame, the frame from the media repository including a bag of objects, said searching resolving the anaphoric ambiguity by replacing the pronoun with an object from the bag of objects A computer system capable of executing a method including

9. Converting the multimedia data into the signal wave is using an auto-decoder neural network to convert the multimedia data into the plurality of amplitudes generating the signal wave from the plurality of amplitudes based on a time frame of each of the plurality of amplitudes The computer system according to claim 8, comprising

10. The computer system according to claim 8, wherein each of the plurality of frames has a duration, and the duration is determined based on a short-time Fourier transform of the signal wave.

11. further comprising separating the signal wave into frames using a spectrogram approach The computer system according to claim 8, comprising

12. The computer system according to claim 11, wherein the highest correlation coefficient is based on the spectrogram approach.

13. Using a deep neural network to identify the label of the frame The computer system according to claim 8, further comprising

14. Generating a vector describing the object, the vector including speed, acceleration, and the label, the generating The computer system according to claim 13, further comprising

15. A computer program for causing a computer to execute the method according to any one of claims 1 to 7

Images