Mapping visual tokens to sound tokens using text similarity
By using machine learning technology and deep learning methods to automatically match video visual features with sound effect tags, the problem of time-consuming sound effect insertion in the computer simulation industry has been solved, achieving efficient sound effect and video content matching and improving game design and production efficiency.
Patent Information
- Application Number
- CN202080032378.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-04-14
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-04-14
AI Technical Summary
In the computer simulation industry, especially in the gaming industry, the process of designing and producing post-video audio effects (SFX) is time-consuming and inefficient, and existing technologies struggle to efficiently match audio effects with video content.
Employing machine learning techniques and deep learning methods, this method automatically recommends and inserts sound effects based on the visual features and sound effect tags of videos. This includes semantic matching of visual tags and sound effect tags, as well as numerical vector similarity calculation. The sound effect tags are trained using supervised and semi-supervised learning models.
It achieves efficient matching of sound effects and video content, improves the automation and accuracy of sound effect insertion, reduces manual intervention time, and enhances game design and production efficiency.
Smart Images

Figure CN113767647B_ABST
Abstract
Description
Technical Field
[0001] The present application generally relates to technically creative and unconventional solutions that are necessarily rooted in computer technology and result in specific technical improvements. Background Art
[0002] Machine learning (sometimes also called deep learning) can be used for a variety of useful applications related to data understanding, detection and / or classification. In the computer simulation industry (such as the gaming industry), video and audio are two separate processes. First, the simulation is designed and produced without audio, and then the audio team surveys the entire simulation video and inserts the corresponding SFX from the sound effects (SFX) database, which is very time-consuming. Summary of the invention
[0003] As understood herein, machine learning may be used to solve the above-mentioned technical problems by providing SFX recommendations relevant to computer simulation scenarios.
[0004] Thus, in one aspect, an apparatus includes at least one processor and at least one computer storage device, the computer storage device being non-transient and including instructions executable by the processor to classify sound effects (SFX) to present the classified SFX. The instructions are also executable to: semantically match SFX tags of the classified SFX with visual tags derived from video recognition of attributes in at least a first video to merge the classified SFX associated with the SFX tags into the first video.
[0005] In some embodiments, the instructions are executable to: generate the visual tag based on recognition of at least one object in the first video and / or based on recognition of at least one action in the first video and / or based on recognition of at least one subtitle in the first video.
[0006] In an exemplary implementation, the instructions are executable to semantically match the SFX tag with the visual tag using textual similarity between the SFX tag and the visual tag.
[0007] In some embodiments, the instructions are executable to: derive a first numerical vector corresponding to the visual label; derive a second numerical vector corresponding to the SFX label; and determine the similarity of the SFX label to the visual label at least in part by calculating a distance between the first vector and the second vector. In such embodiments, the instructions are executable to: in response to determining that a distance between the first vector and a second vector associated with a first SFX label is less than a distance between the first vector and a second vector associated with a second SFX label, determine that the first SFX label is more similar to a visual label than the second SFX label.
[0008] If desired, the instructions are executable to map a single visual tag to multiple SFX tags.
[0009] On the other hand, a method includes: generating at least one visual tag describing at least one attribute of at least a first video; and associating at least one sound effect (SFX) associated with the SFX tag with the first video based at least in part on semantic similarity between the visual tag and at least one SFX tag.
[0010] In another aspect, an assembly includes: at least one computer storage device that is not a transient signal and instead includes instructions executable by at least one processor for identifying at least one visual marker describing at least a first video. The instructions are also executable for associating at least one sound effect (SFX) associated with the SFX marker with the first video based at least in part on a similarity between the visual marker and at least one SFX marker.
[0011] The details of the present application, both as to its structure and operation, may be best understood by reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 is a block diagram of an exemplary system consistent with the principles of the present invention;
[0013] Figure 2 is a flow chart of exemplary overall logic for recommending sound effects (SFX) for a video or computer simulation consistent with the principles of the present invention;
[0014] Figure 3 Yes Description Figure 2 A block diagram of the logic;
[0015] Figure 4 is a block diagram of a first method of SFX tags involving direct mapping from video to SFX tags;
[0016] Figure 5 is in line with Figure 4 A block diagram of additional features of the first method in;
[0017] Figure 6 and Figure 7 is in line with Figure 4 Block diagram of the machine learning architecture associated with “noisy” coarse-grained (in the example shown, 32 classes) SFX classification for the first approach in FIG.
[0018] Figure 8 is in line with Figure 4 Block diagram of the semi-supervised machine learning architecture associated with “noisy” fine-grained SFX classification for the first approach in [1].
[0019] Fig. 9 is in line with Figure 4 A block diagram of the machine learning architecture associated with the training and testing phases of the first approach in ;
[0020] Fig.10 is a block diagram of features of a second approach to video tagging involving indirect tagging through visual understanding; and
[0021] Fig.11 and Fig.12 is a screenshot and associated table illustrating how to associate a visual tag with a matching SFX audio tag. DETAILED DESCRIPTION
[0022] In accordance with the present principles, a deep learning-based domain adaptation approach may be used to recommend SFX for videos and computer simulations, such as video games.
[0023] The method described herein may involve multiple objects and multiple actions associated with the multiple objects. For example, a plurality of image text blocks of text may be "objects", and the type of the image blocks may be "actions".
[0024] The present disclosure also relates generally to a computer ecosystem that includes aspects of a network of consumer electronics (CE) devices, such as, but not limited to, distributed computer gaming networks, augmented reality (AR) networks, virtual reality (VR) networks, video broadcasting, content delivery networks, virtual machines, and artificial neural network and machine learning applications.
[0025] The system herein may include server and client components connected via a network so that data can be exchanged between the client and server components. The client component may include one or more computing devices, including AR headsets, VR headsets, game consoles (such as Sony ) and related motherboards, game controllers, portable televisions (e.g., smart TVs, Internet-enabled TVs), portable computers (such as laptops and tablet computers), and other mobile devices (including smart phones and additional examples discussed below). These client devices can operate in a variety of operating environments. For example, some of the client computers may use, for example, Orbis or Linux operating systems, operating systems from Microsoft, or Unix operating systems, or operating systems produced by Apple or Google. These operating environments can be used to execute one or more programs / applications, such as browsers made by Microsoft or Google or Mozilla, or other browser programs that can access websites hosted by Internet servers discussed below. In addition, an operating environment according to the principles of the present invention can be used to execute one or more computer game programs / applications and other programs / applications that implement the principles of the present invention.
[0026] The server and / or gateway may include one or more processors executing instructions that configure the server to receive and transmit data over a network such as the Internet. Additionally or alternatively, the client and server may be connected via a local intranet or a virtual private network. The server or controller may be provided by a game console and / or one or more motherboards thereof (such as a Sony ), personal computers, etc.
[0027] Information may be exchanged between the client and the server over the network. To this end and for security, the server and / or client may include firewalls, load balancers, temporary storage devices and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus for implementing a method for providing network users with a secure community such as an online social networking site or a video game site to communicate crowdsourcedly in accordance with the principles of the present invention.
[0028] As used herein, instructions refer to computer-implemented steps for processing information in a system. Instructions may be implemented in software, firmware, or hardware, and include any type of programmed steps undertaken by components of the system.
[0029] The processor may be any conventional general purpose single-chip or multi-chip processor that can execute logic through various lines such as address lines, data lines, and control lines, as well as registers and shift registers.
[0030] The software modules described by the flowcharts and user interfaces herein may include various subroutines, programs, etc. Without limiting the present disclosure, the logic stated as being performed by a particular module may be redistributed to other software modules and / or combined together in a single module and / or made available in a sharable library.
[0031] As indicated above, the inventive principles described herein may be implemented as hardware, software, firmware, or a combination thereof; accordingly, the illustrative components, blocks, modules, circuits, and steps are described in terms of their functionality.
[0032] In addition to what has been mentioned above, the logic blocks, modules and circuits described below may be implemented or executed with a general purpose processor, a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices designed to perform the functions described herein (such as application specific integrated circuits (ASICs), discrete gate or transistor logic, discrete hardware components or any combination thereof). The processor may be implemented by a combination of a controller or a state machine or a computing device.
[0033] The functions and methods described below can be implemented with hardware circuits or software circuits. When implemented with software, the functions and methods can be written in appropriate languages such as, but not limited to, Java, C#, or C++, and can be stored on or transmitted by computer-readable storage media, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk read-only memory (CD-ROM), or other optical disk storage devices (such as digital versatile disks (DVD)), magnetic disk storage devices, or other magnetic storage devices including removable thumb drives, etc. Computer-readable media can be established by connection. Such connections can include, for example, hard-wired cables, including optical fibers and coaxial cables, and digital subscriber lines (DSL) and twisted pairs. Such connections can include wireless communication connections, including infrared and radio.
[0034] The components included in one embodiment may be used in other embodiments in any appropriate combination. For example, any of the various components described herein and / or depicted in the figures may be combined, interchanged, or excluded from other embodiments.
[0035] “A system having at least one of A, B, and C” (similarly, “a system having at least one of A, B, or C” and “a system having at least one of A, B, C”) includes the following systems: having only A; having only B; having only C; having both A and B; having both A and C; having both B and C; and / or having A, B, and C, etc.
[0036] Now specifically refer to Figure 1, an exemplary system 10 is shown, which may include one or more of the exemplary devices mentioned above and further described below according to the principles of the present invention. The first of the exemplary devices included in the system 10 is a consumer electronics (CE) device such as an audio video device (AVD) 12, such as but not limited to an Internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). However, the AVD 12 may alternatively be an appliance or household item, such as a computerized Internet-enabled refrigerator, washer, or dryer. Alternatively, the AVD 12 may also be a computerized Internet-enabled ("smart") phone, a tablet computer, a notebook computer, an augmented reality (AR) headset, a virtual reality (VR) headset, Internet-enabled or "smart" glasses, another type of wearable computerized device, such as a computerized Internet-enabled watch, a computerized Internet-enabled bracelet, a computerized Internet-enabled music player, a computerized Internet-enabled head-mounted phone, a computerized Internet-enabled implantable device (such as an implantable skin device), other computerized Internet-enabled devices, etc. Regardless, it should be understood that AVD 12 is configured to implement the present principles (e.g., communicate with other consumer electronics (CE) devices to implement the present principles, execute the logic described herein, and perform any other functions and / or operations described herein).
[0037] Therefore, in order to implement such a principle, the AVD 12 can be composed of Figure 1 Some or all of the components shown are established. For example, the AVD 12 may include one or more displays 14, which may be implemented by a flat screen of high definition or ultra high definition ("4K") or higher, and may be touch-enabled for receiving user input signals via touch on the display. The AVD 12 may include: one or more speakers 16, which are used to output audio according to the principles of the present invention; and at least one additional input device 18 (such as an audio receiver / microphone), which is used to input audible commands to the AVD 12 to control the AVD 12. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating through at least one network 22 (such as the Internet, WAN, LAN, etc.) under the control of one or more processors. Therefore, the interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. In addition, it should be noted that the network interface 20 may be, for example, a wired or wireless modem or router or other suitable interface (such as, for example, a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above, etc.).
[0038] It should be understood that one or more processors control AVD 12 to implement the principles of the present invention, including other elements of AVD 12 described herein, such as controlling display 14 to present images on the display and receive input from the display. One or more processors may include a central processing unit (CPU) 24 and a graphics processing unit (GPU) 25 on a graphics card 25A.
[0039] In addition to the foregoing, the AVD 12 may also include one or more input ports 26, such as, for example, a high-definition multimedia interface (HDMI) port or a USB port for physically connecting (e.g., using a wired connection) to another consumer electronics (CE) device and / or a headphone port for connecting headphones to the AVD 12 for presenting audio from the AVD 12 to the user through the headphones. For example, the input port 26 may be connected to a cable or satellite source 26a of audio-video content via a wire or wireless connection. Thus, the source 26a may be, for example, a separate or integrated set-top box or satellite receiver. Alternatively, the source 26a may be a game console or disk player containing content that may be considered by the user as a favorite for channel allocation purposes. When implemented as a game console, the source 26a may include some or all of the components described below with respect to the CE device 44, and may implement some or all of the logic described herein.
[0040] The AVD 12 may also include one or more computer memories 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices, which are in some cases embodied as independent devices in the housing of the AVD, or as personal video recorders (PVRs) or video disk players for playing back AV programs inside or outside the housing of the AVD, or as removable storage media. In addition, in some embodiments, the AVD 12 may include a position or location receiver (such as, but not limited to, a cell phone receiver, a GPS receiver, and / or an altimeter 30), which is configured to, for example, receive geographic location information from at least one satellite or cell phone tower and provide the information to the processor 24 and / or determine the altitude at which the AVD 12 is set in conjunction with the processor 24. However, it should be understood that another suitable position receiver other than a cell phone receiver, a GPS receiver, and / or an altimeter may be used in accordance with the principles of the present invention, for example, to determine the location of the AVD 12 in all three dimensions.
[0041] Continuing the description of AVD 12, in some embodiments, according to the principles of the present invention, AVD 12 may include one or more cameras 32, which may be, for example, thermal imaging cameras, digital cameras such as webcams, infrared (IR) cameras, and / or cameras integrated into AVD 12 and controllable by processor 24 to generate pictures / images and / or videos. AVD 12 may also include a Bluetooth transceiver 34 and other near field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technology, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.
[0042] Further, the AVD 12 may include one or more auxiliary sensors 37 (e.g., motion sensors such as accelerometers, gyroscopes, gyroscopes or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, gesture sensors (e.g., for sensing gesture commands), etc.) that provide input to the processor 24. The AVD 12 may include a wireless TV broadcast port 38 for receiving OTA TV broadcasts that provide input to the processor 24. In addition to the foregoing, it should be noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the AVD 12.
[0043] Still refer to Figure 1 In addition to the AVD 12, the system 10 may also include one or more other consumer electronics (CE) device types. In one example, a first CE device 44 may be used to send computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or through a server described below, while a second CE device 46 may include similar components to the first CE device 44. In the example shown, the second CE device 46 may be configured as an AR or VR headset worn by a user 47, as shown. In the example shown, only two CE devices 44, 46 are shown, and it should be understood that fewer or larger devices may also be used according to the principles of the present invention.
[0044] In the example shown, it is assumed that all three devices 12, 44, 46 are components of a network (such as a secure or encrypted network, an entertainment network, or Wi-Fi) in a home, for example, or at least appear close to each other in a particular location and are able to communicate with each other and with the server described herein. However, unless otherwise expressly required, the principles of the present invention are not limited to a particular location or network.
[0045] An exemplary, non-limiting first CE device 44 may be established by any of the above-mentioned devices (e.g., a smart phone, a digital assistant, a portable wireless laptop or notebook computer, or a game controller (also referred to as a "console")), and may therefore have one or more of the components described below. The second CE device 46 may be established by, but is not limited to, an AR headset, a VR headset, "smart" Internet-enabled glasses, or even a video disc player (such as a Blu-ray player), a game console, etc. Further, in some embodiments, the first CE device 44 may be a remote control device (RC) for, for example, issuing AV play and pause commands to the AVD 12, or it may be a more complex device, such as a tablet computer, a computer connected to a computer via a wired or wireless link. Figure 1 Another of the illustrated devices implements a game console that communicates with and controls a game controller, personal computer, wireless telephone, or the like, for video game presentation on AVD 12 .
[0046] Thus, the first CE device 44 may include one or more displays 50, which may be touch-enabled for receiving user input signals via touch on the display 50. Additionally or alternatively, the one or more displays 50 may be at least partially transparent displays configured for presenting AR and / or VR images, such as AR headset displays or "smart" glasses displays or "heads-up" displays, as well as VR headset displays or other displays.
[0047] The first CE device 44 may also include one or more speakers 52 for outputting audio according to the principles of the present invention, and at least one additional input device 54, such as, for example, an audio receiver / microphone, for inputting audible commands to the first CE device 44 to control the device 44. The exemplary first CE device 44 may also include one or more network interfaces 56 for communicating through the network 22 under the control of one or more CE device processors 58. Thus, the interface 56 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, including a mesh network interface. It should be understood that the processor 58 controls the first CE device 44 to implement the principles of the present invention, including other elements of the first CE device 44 described herein, such as, for example, controlling the display 50 to present images on the display and receiving input from the display. In addition, it should be noted that the network interface 56 may be, for example, a wired or wireless modem or router or other suitable interface (such as a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above, etc.).
[0048] Further, it should be noted that the first CE device 44 may include a graphics processing unit (GPU) 55 on a graphics card 55A in addition to one or more processors 58. The graphics processing unit 55 may be configured to, among other things, render AR and / or VR images on the display 50.
[0049] In addition to the foregoing, the first CE device 44 may also include one or more input ports 60 (such as, for example, an HDMI port or a USB port) for physically connecting (e.g., using a wired connection) to another CE device and / or a headphone port for connecting headphones to the first CE device 44 for presenting audio from the first CE device 44 to the user through the headphones. The first CE device 44 may also include one or more tangible computer-readable storage media 62, such as a disk-based storage device or a solid-state storage device. In addition, in some embodiments, the first CE device 44 may include a position or location receiver (such as, but not limited to, a cell phone and / or a GPS receiver and / or an altimeter 64) configured to receive geographic location information from at least one satellite and / or cell phone tower using, for example, triangulation and provide the information to the CE device processor 58 and / or determine, in conjunction with the CE device processor 58, the altitude at which the first CE device 44 is located. However, it should be understood that another suitable position receiver other than a cell phone and / or a GPS receiver and / or an altimeter may be used, for example, to determine the location of the first CE device 44 in all three dimensions in accordance with the principles of the present invention.
[0050] Continuing with the description of the first CE device 44, in some embodiments, in accordance with the principles of the present invention, the first CE device 44 may include one or more cameras 66, which may be, for example, thermal imaging cameras, IR cameras, digital cameras such as webcams, and / or another type of camera integrated into the first CE device 44 and controllable by the CE device processor 58 to generate pictures / images and / or videos. A Bluetooth transceiver 68 and other near field communication (NFC) elements 70 may also be included on the first CE device 44 for communicating with other devices using Bluetooth and / or NFC technology, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.
[0051] Further, the first CE device 44 may include one or more auxiliary sensors 72 (e.g., motion sensors such as accelerometers, gyroscopes, gyroscopes or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, gesture sensors (e.g., for sensing gesture commands), etc.) that provide input to the CE device processor 58. The first CE device 44 may include other sensors that provide input to the CE device processor 58, such as, for example, one or more climate sensors 74 (e.g., barometers, humidity sensors, wind sensors, light sensors, temperature sensors, etc.) and / or one or more biometric sensors 76. In addition to the foregoing, it should be noted that in some embodiments, the first CE device 44 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 78, such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the first CE device 44. The CE device 44 may communicate with the AVD 12 via any of the above-mentioned communication modes and related components.
[0052] The second CE device 46 may include some or all of the components shown for the CE device 44. Either or both CE devices may be powered by one or more batteries.
[0053] Referring now to the aforementioned at least one server 80, it includes at least one server processor 82, at least one tangible computer readable storage medium 84 (such as a disk-based storage device or a solid-state storage device). In an implementation, the medium 84 includes one or more solid-state storage drives (SSDs). In accordance with the principles of the present invention, the server also includes at least one network interface 86, which allows communication with the server via the network 22. Figure 1 The server 80 may be configured to communicate with other devices of the server and may actually facilitate communication between the server and the client devices. It should be noted that the network interface 86 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface (such as a wireless telephone transceiver). The network interface 86 may be a remote direct memory access (RDMA) interface that directly connects the medium 84 to a network such as a so-called "fabric" without going through the server processor 82. The network may include an Ethernet network and / or a Fibre Channel network and / or an Infiniband network. Typically, the server 80 includes multiple processors in multiple computers, referred to as "blades" that may be arranged in a physical server "stack".
[0054] Thus, in some embodiments, server 80 may be an Internet server or an entire "server farm" and may include and perform "cloud" functions, such that devices of system 10 may access a "cloud" environment via server 80 in, for example, an exemplary embodiment of domain adaptation as disclosed herein. Additionally or alternatively, server 80 may be provided by a server associated with Figure 1 The other devices shown are implemented in the same room or nearby one or more game consoles or other computers.
[0055] Figure 2 and Figure 3 The overall principle is shown. Figure 2 Beginning at block 200 in the example, sound effects (SFX) are classified. In one example, this classification can be performed on an incoming digitized sound effect signal 300 to present a tag 302 (in a textual description of the classified sound effect as described elsewhere herein) Figure 3 ).
[0056] Move to Figure 2 block 202 and still cross-reference Figure 3 , the tag 302 is recorded in the database 304. Then, proceed to Figure 2 In block 204, the recorded markup can be combined with the video without sound 306 to present a video with sound effects 308. It should be noted that "sound effects" refers to non-verbal audio that is part of a computer simulation (such as a computer game) to imitate the sound of gunfire, the sound of fire burning, the sound of people running, the sound of people shouting, the sound of water, etc. As further explained below, deep learning / AI techniques are provided herein to help create sound content for computer simulations (such as video games).
[0057] As used herein, "clean SFX tagging" refers to categorizing or tagging clean audio samples (sound effects with a single sound source) used by game sound designers based on their categories and subcategories so that they can be automatically recorded in a database. This helps game designers search and retrieve more efficiently during mixing. "Video tagging" refers to automatically recommending sound effects related to game scenes. This is done to help game designers by making the sound design process more efficient. The present principles focus on techniques for implementing video tagging.
[0058] This disclosure discloses two techniques for video tagging. Figures 4 to 9 A direct mapping approach is described in which a deep learning engine is trained to learn correlations between visual features of a game video and corresponding SFX (audio) tags 302. Figures 10 to 12 A visual understanding method is described in two steps, namely providing a neural network (NN) to understand the visual content of the game scene and generate visual tags, including object tags, action tags, and subtitles, and then mapping the visual tags to audio tags using semantic textual similarity. Dictionary-based mapping can also be used based on other knowledge bases.
[0059] Therefore, and now with reference to Figure 4, in a training phase 400, a video (such as a computer simulation using SFX sounds 402) is used to train the NN system to generate labels 404 for different SFX sources, thereby rendering SFX labels 406. Once the NN system is trained, it can be used in a testing phase 408 to receive a video 410 (such as a computer simulation without SFX sounds) as input to a training model 412 described further below to output SFX labels 414 combined with the video 410, thereby rendering a video 416 in which the SFX sounds are incorporated.
[0060] Reference now Figure 5 , you can see Figure 4 A more detailed explanation of the training phase in . A silent video (such as a computer game video 500) is input to a trained NN 502. The NN 502 uses a supervised learning method to learn a direct mapping between the visual features of the video and the corresponding sound effects. In order to train this supervised model, sound annotations of the game audio are required. As understood herein, since game audio typically contains a mixture of sounds (also known as noisy SFX), it is difficult to obtain human annotations 504, especially when the number of sound categories is large, so the process becomes complicated. Therefore, a deep learning model 506 is trained to automatically label the mixture of sounds (noisy SFX model) to identify the categories of the constituent sounds.
[0061] Reference now Figure 6 and Figure 7 In an initial implementation, a noisy SFX model is trained to label a small number of categories (32 categories) using human annotations. The audio clip represented by the spectrogram 600 is input to a segmentation mapping module 602 comprising a series of convolutional NNs (CNNs) 604. The segmentation mask 606 is output by the mapping module 602 and used in a classification map 608, which produces a prediction 610 for the label with a corresponding probability. Figure 7 Relatedly shown is a gated convolutional recurrent NN (CRNN) 700 that receives an SFX clip 702 as input and extracts spectral patterns at each time step, providing output to a bidirectional RNN 704, such as a bidirectional long short-term memory (LSTM). Figure 7 Indicates the type of CNN used in network 700. Bi-RNN 704 is coupled to an attention-based localization module 706, which includes multiple feed-forward NNs (FNNs) operating as sigmoid and SoftMax FNNs as shown to produce a predicted label 708 as a weighted average.
[0062] Important to the principle of the present invention is Figure 8, showing an advanced technique for noisy SFX labeling. In order to generate finer-grained SFX labels (e.g., 182-class or 5000-class labels or even more detailed) to better distinguish different sound effects in noisy samples, a supervised model is trained using actual SFX data 800 from computer simulation and synthetic noisy SFX data 802 generated separately from any simulation, only for the purpose of training the gated CNN module 804. In other words, Figure 8 The inventive principles reflected in recognize that to train a supervised model, training data with more fine-grained (e.g., 182 or 5000 categories) ground truth labeled training data is required, while for currently available game videos, there are only coarse-grained (32 categories) human-annotated SFX labels. Figure 8 The following figure shows a semi-supervised method that generates fine-grained audio tags from coarse-grained audio tags without additional human annotation. It should be noted that 32 classes and 182 classes are used as examples of coarse-grained and finer-grained tags.
[0063] A synthetic mixture of the sound samples represented at synthetic noisy SFX data 802 is created and their categories are recorded during mixing. In this synthetic mixture, fine-grained SFX labels are established (referred to as Dataset 1 elsewhere in this document). On the other hand, block 800 represents available real game audio with coarse-grained labels (generated by humans) (referred to as Dataset 2). Figure 8 As shown, a dataset 1 of actual simulation or game data and a dataset 2 of synthetic data that is not from simulation or game but is created for the purpose of supplementing game data are combined to train an end-to-end semi-supervised model 804, which includes a coarse classifier 806 and a fine-grained classifier 808 to generate fine-grained labels 810 that identify the components of noisy game audio. It is semi-supervised because there are no real fine-grained game audio labels for training, as explained earlier. It is a multi-task model because it is able to generate coarse-grained audio labels 812 and fine-grained audio labels 810. In other words, fine-grained analysis uses more categories than coarse-grained analysis.
[0064] The training loss function for this model is the sum of the losses of the fine-grained and coarse-grained labels. The goal of training is to minimize the training loss. Training stops when the model converges. At this point, a model is obtained that can decompose a noisy audio mixture into its component labels.
[0065] Thus, the above description discloses a technique for identifying the component sound effect categories of game audio, and Fig. 9 Describes how these tags are used (by humans or Figure 8As shown in the figure, during training, a video 900 with sound extracted is input to a training phase module 904 along with noisy SFX tags 902 generated and / or annotated by humans as described above. More specifically, the corresponding audio extracted from the video is generated by the above in Figure 8 The noisy SFX model explained in 900 is used to generate SFX tags (or labels) 902, which are input to the supervised training stage model 904 along with the corresponding video segmentation 900. In this way, the video is synchronized with the audio tags before training. In an exemplary non-limiting implementation, the frame rate used can be thirty frames per second (30fps) and the video duration can be one second.
[0066] The training phase module 904 generates video embeddings (numeric vectors) by passing silent video frames through a deep CNN 906 (e.g., Resnet or similar network). For each frame, an embedding (vector) is generated, which serves as a visual feature of the video frame. Other visual features can also be used. Because the video is a sequence of frames, a sequence of video embeddings is generated, which is then input to a recurrent neural network 908, in the example shown, a bidirectional gated recurrent unit (GRU) or gated recurrent network that generates a label prediction 910.
[0067] The output of the training is a neural model 912 that can receive new simulated videos without sound 914 during the testing phase and generate sound tags 916 corresponding to the silent videos 914. These tags can be used to retrieve corresponding sound effects 918 to combine with the video, as shown at 920.
[0068] Figures 10 to 12 The visual understanding method mentioned above is shown. In the first step, a video 1000 (such as a computer simulation without sound (audio)) is used to generate visual tags 1002 based on visual understanding of, for example, objects 1004 identified in the video, actions 1006 identified in the video, and scene descriptions 1008 identified in the video. Then, a semantic text similarity module 1010 receives the visual tags 1002 and SFX tags 1012 from the database described above to automatically map the visual tags to specific audio categories in the sound database to generate a video with sound 1014.
[0069] Fig.11To further illustrate. Display 1100 is shown presenting a video with an object 1102 identified using image recognition techniques to generate a corresponding visual tag 1104. Visual tags 1104 may be embedded using word embeddings or sentence embeddings, which produces a numerical vector. Visual tags 1104 are matched with corresponding audio tags 1106. Each audio category or audio file name of the identified audio sample is embedded using word embeddings or sentence embeddings, which in turn produces a numerical vector. Fig.12 Similarly shown is a video on display 1200 with subtitles 1202 which may be matched to audio tags 1204 using an unsupervised semantic textual similarity model 1203 .
[0070] In any case, whether matching audio tags with object tags, subtitle tags, or action tags, two numeric vectors are produced, one for the audio tags and the other for the tags derived from the video. The similarity of the tags is determined by calculating the distance between the two vectors. Any distance metric can be used, such as cosine similarity or Euclidean distance. The smaller the distance, the more similar the tags. Using this method, each visual tag is mapped to the top k most similar audio tags.
[0071] Using this text similarity approach, each visual token can be mapped to an audio token of varying granularity, ranging from a coarse-grained (e.g., 32-class) token identifying a group of audio samples to a very fine-grained token identifying a single sound sample.
[0072] The audio tags automatically generated from the visual understanding of the game scene can be used for two purposes. First, the audio tags can be used to recommend sound effects for the game scene to the game designer. Second, the audio tags can also be used as SFX labels for training Figures 4 to 12 The direct-mapped video labeling model disclosed in [5] is an alternative to noisy SFX labels derived from audio.
[0073] Although Figures 4 to 9 Direct mapping in Figures 10 to 12 The visual understanding technique shown in , however, presents more fine-grained labeling using unsupervised text similarity and makes it relatively easy to annotate objects and subtitles. Direct mapping is particularly advantageous when accurate ground truth SFX labeling is available for labeling or sound source separation is feasible. Visual understanding is particularly advantageous when obtaining fine-grained SFX annotations is otherwise difficult, and it mimics the workflow of sound engineers.
[0074] The inventive principle can be used in deep learning-based methods for image, video and audio data processing, etc. As can be understood from the previous detailed description, the inventive principle thus improves the adaptation and training of neural networks through the technical solutions described herein.
[0075] It will be appreciated that while the principles of the invention have been described with reference to certain exemplary embodiments, these embodiments are not intended to be limiting and that various alternative arrangements may be used to implement the subject matter claimed herein.
Claims
1. A device for recommending sound effects (SFX) for a video, comprising: At least one processor is programmed with instructions to: Presenting categorized SFX; as well as semantically matching SFX tags of the classified SFX with visual tags derived from video recognition of attributes in at least a first video to merge the classified SFX associated with the SFX tags into the first video, wherein the instructions are executable to: The SFX labels are matched to the visual labels using a computer-implemented model trained using synthetic fine-grained SFX labels and coarse-grained SFX labels obtained from actual computer simulations, wherein the synthetic fine-grained SFX labels are generated separately from any simulation, and the fine-grained SFX labels have more categories than the coarse-grained SFX labels.
2. The apparatus of claim 1, wherein the instructions are executable to: The visual tag is generated based on recognition of at least one object in the first video.
3. The apparatus of claim 1 , wherein the instructions are executable to: The visual tag is generated based on recognition of at least one action in the first video.
4. The apparatus of claim 1, wherein the instructions are executable to: The visual tag is generated based on identification of at least one subtitle in the first video.
5. The apparatus of claim 1, wherein the instructions are executable to semantically match the SFX tag with the visual tag using textual similarity between the SFX tag and the visual tag.
6. The apparatus of claim 1, wherein the instructions are executable to: deriving a first vector corresponding to the visual label; deriving a second vector corresponding to the SFX label; and The similarity of the SFX label to the visual label is determined at least in part by calculating a distance between the first vector and the second vector.
7. The apparatus of claim 6, wherein the instructions are executable to: In response to determining that a distance between the first vector and a second vector associated with a first SFX label is less than a distance between the first vector and a second vector associated with a second SFX label, the first SFX label is determined to be more similar to a visual label than the second SFX label.
8. The apparatus of claim 6, wherein the instructions are executable to: Map a single visual label to multiple SFX labels.
9. A method for recommending sound effects (SFX) for a video, comprising: generating at least one visual indicia describing at least one attribute of at least a first video; as well as associating at least one SFX associated with the SFX marker with the first video based at least in part on semantic similarity between the visual marker and at least one SFX marker, wherein The visual tags are associated with corresponding visual vectors, the SFX tags are associated with corresponding SFX vectors, and the semantic similarity is identified at least in part by calculating a distance between at least one visual vector and at least one SFX vector, wherein a smaller distance indicates a higher match between the video and the SFX, such that each visual tag is mapped to the top k most similar SFX tags.
10. The method of claim 9, wherein the attribute comprises an object. The method of claim 9 , wherein the attribute comprises an action.
12. The method of claim 9, wherein the attributes include subtitles. The method of claim 9 , wherein the distance comprises a cosine similarity distance. The method of claim 9 , wherein the distance comprises a Euclidean distance.
15. The method of claim 9, comprising: deriving a first vector corresponding to the visual marker; deriving a second vector corresponding to the SFX marker; as well as The similarity of the SFX marker to the visual marker is determined at least in part by calculating a distance between the first vector and the second vector.
16. The method of claim 15, comprising: In response to determining that a distance between the first vector and a second vector associated with the first SFX marker is less than a distance between the first vector and a second vector associated with the second SFX marker, the first SFX marker is determined to be more similar to the visual marker than the second SFX marker.
17. The method of claim 9, comprising: Map a single visual marker to multiple SFX markers.
18. A component for recommending sound effects (SFX) for a video, comprising: At least one computer storage device, the at least one computer storage device comprising instructions executable by at least one processor for: identifying at least one visual indicia describing at least a first video; and associating at least one SFX associated with the SFX marker with the first video based at least in part on a similarity between the visual marker and at least one SFX marker, wherein The visual tags are associated with corresponding visual vectors, the SFX tags are associated with corresponding SFX vectors, and the similarity is identified at least in part by calculating a distance between at least one visual vector and at least one SFX vector, wherein a smaller distance indicates a higher match between the video and the SFX, such that each visual tag is mapped to the top k most similar SFX tags.
19. The assembly of claim 18, comprising the processor.
20. The component of claim 18, wherein the instructions are executable to: deriving a first vector corresponding to the visual marker; deriving a second vector corresponding to the SFX marker; and The similarity of the SFX marker to the visual marker is determined at least in part by calculating a distance between the first vector and the second vector.
21. The component of claim 20, wherein the instructions are executable to: In response to determining that a distance between the first vector and a second vector associated with the first SFX marker is less than a distance between the first vector and a second vector associated with the second SFX marker, the first SFX marker is determined to be more similar to the visual marker than the second SFX marker.
Citation Information
Patent Citations
Systems, methods, and apparatus for generating an audio-visual presentation using characteristics of audio, visual and symbolic media objects
US20150234833A1
