Low latency description system

Abstract

An artificial intelligence (AI) low latency processing system is provided. The low latency processing system includes: a processor; and a memory storing instructions. The low latency processing system is configured to: collect a sequence of frames, the sequence of frames jointly including information dispersed in at least some frames of the sequence of frames; executing a timed neural network trained to identify an early frame sub-sequence of the frame sequence comprising at least a portion of information indicative of the information; and executing a decoding neural network trained to decode the information from a portion of the information in a frame sub-sequence, where the timing neural network and the decoding neural network are jointly trained to decode the information from the portion of the information in the frame sub-sequence. To iteratively identify a minimum number of subframes starting from the start of a sequence of training frames containing a portion of training information sufficient to decode the training information.

Description

Technical Field

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS)

[0002] This application is a continuation-in-part of patent application serial number 17 / 384,234 filed on July 23, 2021, the contents of which are incorporated herein by reference.

[0003] The present invention generally relates to captioning and interaction based on captioning, and more particularly, to end-to-end captioning with delay optimization capabilities. Background Art

[0004] At any moment, countless events happening in the real world are captured by sensors such as cameras, microphones, LiDAR, mmWave and stored as massive sensor data resources. To efficiently retrieve these records, scene description is a fundamental technology in both offline and online settings, thanks to its ability to understand the scene and describe the events in natural language. In addition, robots can interact with humans or other robots to decide the next action based on scene understanding through online scene description.

[0005] Scene description techniques have been actively studied in the field of computer vision for video description. Deep recurrent neural networks (RNNs) have been applied to video description, where RNNs are trained to convert sequences of image features extracted from video clips into sequences of words describing the video content. Its goal is to generate video descriptions (annotations) about objects and events in any video clip. Recently, the Transformer model has become increasingly popular as a model for video description over RNNs because of its improved description performance.

[0006] In addition, not only image features but also audio features are extracted from video clips and used to improve description quality, where attention-based multimodal fusion technology has been introduced to effectively fuse image features and audio features according to video content (U.S. Patent 10,417,498).

[0007] Existing methods for video description are essentially assumed to work in an offline manner, where each video clip is given before being described, so the system has access to all frames of the video clip to generate the description. However, these existing methods are not practical in real-time monitoring, surveillance systems, scene-understanding-based interactive systems for car navigation, and robots, where it is crucial to not only accurately describe events, but also generate descriptions as quickly as possible to quickly discover and report events to take the next action. Therefore, low-latency descriptions for online systems are needed to achieve this functionality, where the system needs to decide on the appropriate timing to generate the correct description using only the limited number of frames that the system has received so far. In addition, this low-latency description functionality allows the robot to decide on the next action as quickly as possible to interact with humans and other robots. Summary of the invention

[0008] Some embodiments of the present disclosure are based on the recognition that scene captioning is a basic technology for understanding scenes and describing events in natural language. In order to apply it to real-time monitoring systems, the system not only needs to accurately describe events, but also needs to generate descriptions as quickly as possible. Low-latency descriptions are needed to achieve this function, but this research area of ​​online scene descriptions has not been explored. This paper proposes a new method to optimize the output timing of each description based on the trade-off between latency and description quality. The audio-visual Transformer is trained to generate factual descriptions using only a small part of all scenes, and imitates the output of a pre-trained Transformer that gives all scenes. A CNN-based timing detector is also trained to detect the appropriate output timing at which the descriptions generated by two Transformers become close enough to each other. Using the jointly trained Transformer and timing detector, a description can be generated at an early stage of the event once the event occurs or when it can be predicted.

[0009] An object of some embodiments of the present invention is to provide an end-to-end scene description system and method that can be used in online / offline monitoring systems and scene-aware interactions to understand events using natural language while the system identifies the events as quickly as possible.

[0010] The present disclosure includes a low-latency scene description system that is trained to optimize the output timing of each description based on a trade-off between latency and description quality. The present invention can train a low-latency description generator according to the following strategy: (1) using a low-latency description generator that only sees a small portion of all scenes acquired by sensors or a small portion of all signals to generate factual descriptions; (2) imitating the output of a pre-trained description generator that uses the entire scene to generate output; and (3) training a timing detector that finds the optimal timing of the output description so that the description ultimately generated by the low-latency description generator becomes sufficiently close to its factual description or the description generated by the pre-trained description generator using the entire scene. The low-latency description generator based on (1) and (2) and the timing detector of (3) are jointly trained.

[0011] The jointly trained low-latency description generator and timing detector can generate descriptions at an early stage of a scene acquired by a sensor once an event occurs. In addition, the framework can be applied to predict future events in low-latency descriptions. In addition, by combining multimodal sensing information, events can be identified at an earlier timing triggered by the earliest prompt in one of the modalities without waiting for other prompts in other modalities. In particular, an audio-visual Transformer constructed as a low-latency description generator according to an embodiment of the present invention can be used to generate descriptions based on the timing of audio prompts earlier than the timing of visual prompts. This low-latency video description system using multimodal sensing information can not only help to quickly retrieve events, but also help to respond to scenes early.

[0012] Some embodiments are based on the recognition that experiments using the ActivityNet description dataset show that a system based on the present invention achieves 94% of the upper limit description quality given by a pre-trained Transformer using the entire video clip using only 28% of the frames from the beginning.

[0013] According to some embodiments of the present invention, a scene description system may be provided. In this case, the scene description system may include: an interface configured to obtain a signal stream captured by a multimodal sensor to describe a scene; a memory storing a computer executable scene description model including a multimodal sensor feature extractor, a multimodal sensor feature encoder, a timing detector and a scene description decoder, wherein the multimodal sensor feature encoder is shared by the timing detector and the scene description decoder; a processor connected to the memory, configured to perform the following steps: extracting multimodal sensor features from multimodal sensor signals using a multimodal sensor feature extractor; encoding multimodal sensor features using a multimodal sensor feature encoder; determining the timing of generating a scene description using a timing detector, wherein the timing is arranged at an early stage of the multimodal sensor signal stream; and generating a scene description describing an event based on the multimodal sensor features by using the scene description decoder according to the timing.

[0014] In addition, some embodiments of the present invention are based on the recognition that a computer-executable training method is provided for training a multimodal sensor feature encoder, a timing detector, and a scene description decoder. The method may include the following steps: providing a training data set including a set of multimodal sensor signals and a set of factual scene descriptions; using a feature extractor to convert the multimodal sensor signals into a feature vector sequence; using a future frame eliminator to eliminate future frames from the feature vector sequence, wherein the future frame eliminator obtains a first feature vector from the feature vector sequence and removes other feature vectors to generate a future elimination feature vector sequence; encoding the future elimination feature vector sequence into a hidden activation vector sequence; training a low-latency scene description generator by calculating a loss value; calculating a loss value based on a posterior probability distribution and a factual description; and training a timing detector by calculating a supervisory signal based on the scene description similarity between the scene description of the future elimination feature vector sequence and the factual description.

[0015] In addition, when the scene description system is configured as a video description system, the video description system may include: an interface configured to obtain an audio-visual signal stream including image data and sound data; a memory storing a computer-executable video description model including an audio-visual feature extractor, an audio-visual feature encoder, a timing detector and a video description decoder, wherein the audio-visual encoder is shared by the timing decoder and the timing detector and the video description decoder; a processor connected to the memory, configured to perform the following steps: extracting audio features and visual features from the audio-visual signal using the audio-visual extractor; encoding the audio features and visual features from the audio-visual signal using the audio-visual encoder; determining the timing of generating a video description using the timing detector, wherein the timing is arranged at an early stage of the audio-visual signal stream; and generating a video description describing the audio-visual scene based on the audio features and visual features by using the video description decoder according to the timing.

[0016] According to some embodiments of the present invention, an artificial intelligence (AI) low-latency processing system is provided. The low-latency processing system may include: a processor; and a memory storing instructions, which, when executed by the processor, cause the low-latency processing system to: collect a frame sequence, the frame sequence jointly including information dispersed in at least some frames in the frame sequence; execute a timing neural network, the timing neural network is trained to identify an early frame subsequence in the frame sequence including at least part of the information indicating the information; execute a decoding neural network, the decoding neural network is trained to decode the information from the part of the information in the frame subsequence, wherein the timing neural network and the decoding neural network are jointly trained to iteratively identify the minimum number of subframes starting from the beginning of the training frame sequence that contain part of the training information sufficient to decode the training information.

[0017] In addition, an embodiment of the present invention provides a computer-implemented method for an artificial intelligence (AI) low-latency processing system, the low-latency processing system comprising a processor and a memory, the memory storing instructions of the computer-implemented method for executing steps using the processor. The computer-implemented method may include the following steps: collecting a frame sequence, the frame sequence jointly including information dispersed in at least some frames in the frame sequence; executing a timing neural network, the timing neural network is trained to identify an early frame subsequence in the frame sequence including at least part of information indicating the information; and executing a decoding neural network, the decoding neural network is trained to decode the information from the part of information in the frame subsequence, wherein the timing neural network and the decoding neural network are jointly trained to iteratively identify the minimum number of subframes starting from the beginning of the training frame sequence that contain part of the training information sufficient to decode the training information.

[0018] The presently disclosed embodiments will be further described with reference to the accompanying drawings. The drawings shown are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] [ Figure 1 ]

[0020] Figure 1 is a schematic diagram illustrating a low-delay video description process according to an embodiment of the present invention.

[0021] [ Figure 2 ]

[0022] Figure 2 is a block diagram illustrating a low-latency video description system including a low-latency description generator, a pre-trained description generator, and a timing detector according to an embodiment of the present invention.

[0023] [ Figure 3 ]

[0024] Figure 3 is a block diagram illustrating the training process of a low-latency description generator and a timing detector according to an embodiment of the present invention, which utilizes a pre-trained description generator to jointly optimize the low-latency description generator and the timing detector.

[0025] [ Figure 4 ]

[0026] Figure 4 is a schematic diagram showing an audio-visual Transformer constructed as a low-delay description generator combined with a timing detector for low-delay video description according to an embodiment of the present invention.

[0027] [ Figure 5 ]

[0028] Figure 5 are evaluation results obtained by performing a low-delay video description test using an audio-visual Transformer constructed as a low-delay description generator combined with a timing detector according to an embodiment of the present invention.

[0029] [ Figure 6 ]

[0030] Figure 6 A model architecture consisting of a question encoder, an AV encoder, a timing detector, and a response decoder according to an embodiment of the present invention is shown.

[0031] [ Figure 7 ]

[0032] Figure 7 The video stream according to an embodiment of the present invention is shown, wherein the video has been s Start and e End, this is in line with the fact that the answer is Y e ’ is associated.

[0033] [ Figure 8 ]

[0034] Figure 8 Processing steps are shown that are performed by a low latency processing system using a processor and a memory or memory storage device storing instructions according to one embodiment of the present invention.

[0035] [ Fig. 9 ]

[0036] Fig. 9 The relationship between the delay ratio and the answer accuracy in MSRVTT-QA according to an embodiment of the present invention is shown.

[0037] [ Fig.10 ]

[0038] Fig.10 A comparison of the quality of answer sentences in the AVSD task according to an embodiment of the present invention is shown.

[0039] [ Fig.11A ]

[0040] Fig.11A The distribution of QA over delay in the AVSD-DSTC7 task at a detection threshold F=0.3 according to an embodiment of the present invention is shown.

[0041] [ Fig. 11B ]

[0042] Fig. 11B The distribution of QA over delay in the AVSD-DSTC7 task at a detection threshold F=0.4 according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0043] Although the above-mentioned drawings illustrate the embodiments of the present disclosure, other embodiments are also conceivable as described in the discussion. The present disclosure presents exemplary embodiments by representation and not limitation. Those skilled in the art may design numerous other modifications and embodiments that fall within the scope and spirit of the principles of the embodiments of the present disclosure.

[0044] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the following description of exemplary embodiments will provide a feasible description for implementing one or more exemplary embodiments to those skilled in the art. It is contemplated that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the disclosed subject matter set forth in the appended claims.

[0045] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be appreciated by those skilled in the art that the embodiments may be practiced without these specific details. For example, the systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in terms of unnecessary details. In other cases, known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments. In addition, similar reference numerals and designations in the various drawings indicate similar elements.

[0046] In addition, various embodiments may be described as processes, which are depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe an operation as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps that are not discussed or included in the accompanying drawings. In addition, not all operations in any specifically described process may appear in all embodiments. A process may correspond to a method, a function, a program, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the function may correspond to the function returning to the calling function or the main function.

[0047] In addition, embodiments of the disclosed subject matter may be implemented at least in part manually or automatically. Manual or automatic implementation may be performed or at least assisted by the use of a machine, hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the desired tasks may be stored in a machine-readable medium. A processor may perform the desired tasks.

[0048] The modules and networks illustrated in the present disclosure may be computer programs, software or instruction codes that can execute instructions using one or more processors. The modules and networks may be stored in one or more storage devices or in a computer-readable medium such as a storage medium, a computer storage medium or a data storage device (removable and / or non-removable), such as a disk, an optical disk or a tape, where the computer-readable medium may be accessed from one or more processors to execute instructions.

[0049] Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media may be RAM, ROM, EEPROM or flash memory, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by applications, modules or both using one or more processors. Any such computer storage medium may be part of the device, or accessible or connectable to the device. Any application or module described herein may be implemented using computer-readable / executable instructions that can be stored or preserved by such computer-readable media.

[0050] (Low latency description)

[0051] According to some embodiments of the present disclosure, the scene description system may provide low-latency scene description from real-time monitoring of the scene or online / offline scene streaming to understand the scene and describe events in natural language.

[0052] In some cases, the scene description system may be configured as a video description system, and the video description system may be referred to as a low-latency video description method / system.

[0053] In the following, a low-latency video description method / system is provided as an example description of a low-latency scene description method / system.

[0054] For example, a signal stream obtained by a unimodal sensor or a multimodal sensor can be converted into a description using natural language. When a single sensor is applied, multimodality includes unimodality. Multimodal sensors capture information such as visual images, audio signals, 3D positioning, infrared thermal imaging, Wi-Fi, etc. In some cases, the sensor can be an image sensor, a microphone, an audiovisual signal sensor, LiDAR, mmWave, a thermal sensor, an odor sensor, a tactile sensor, or any combination thereof.

[0055] When the low-latency scene description method / system is applied to a real-time or offline video stream, the multimodal sensor signal can be an audio-video signal stream, the scene encoder can be an audio-visual encoder, and the single-modal / multimodal sensor features can be audio and visual features. In addition, the multimodal sensor signal can be a signal / information obtained by a multimodal sensor. For example, the signal can be any sensing information, such as video frames, sound data, 3D positioning data, thermal sensor data, odor sensor data, tactile sensor data, or any combination thereof.

[0056] Figure 1 1 is a schematic diagram illustrating a low-latency video description process according to an embodiment of the present invention. Assume that a video stream 101 is given as X, in which an event occurs between time s and time T, wherein a fact description 102 as Y for the event may be "one of the workers was hit by a bulldozer". The description is usually a sentence describing the event in natural language. A video description system based on the prior art uses all frames x of the video stream X to generate a video description of the event. s:T Generate this description for a video clip, where x s:T Represents a video frame sequence x s ,x s+1 ,…,x T-1 ,x T , which each correspond to a video frame at each time index. Therefore, these existing video description systems are assumed to work only in an offline manner.

[0057] However, these existing methods are impractical in real-time monitoring or surveillance systems, where it is crucial to not only accurately describe the event, but also to generate the description as quickly as possible. To this end, the low-latency video description module 110 needs to have a timing detector 103 that uses a portion of the video frame 104, i.e., x s:t(video frames from start time s to current time t) determines whether current time t is a suitable timing for outputting a description. Only when timing detector 103 detects a suitable timing, low-latency description generator 105 generates description 106 based on partial video frames 104. The timing of the previous description generated by the system or the timing of detecting a certain change in pixel intensity in the video ( Figure 1 (not shown) to determine the start time s.

[0058] A remaining problem is that there is not enough dataset to train the timing detector 103 and the low-latency description generator 105, where the dataset should be annotated with appropriately early timing and its description for various videos. There are datasets that are only used for offline video description, that is, datasets are annotated with the timing that the event has ended in the video. Usually, it is very expensive to calibrate a large number of new datasets with these early timing annotations.

[0059] To address this problem, an embodiment of the present invention uses only a dataset for offline video description to train a timing detector 103 and a low-latency description generator 105, wherein the detector and the generator are optimized to not only accurately describe the event, but also to generate the description as quickly as possible. Therefore, low-latency video description is achieved by detecting the appropriate timing to generate a correct description using only a portion of the video frames 104 up to the current time t. The system can generate a description 106, which is equal to the fact 102 "one of the workers was hit by a bulldozer", although it only uses a portion of the video frame 104. On the other hand, the existing method for training a generator for offline video description may generate an incorrect description 107 "the worker is walking" for the partial video frame 104, because the generator is trained to generate descriptions about events occurring in each video clip. In the example of low-latency video description 100, the event "one of the workers was hit by a bulldozer" has not yet occurred at time t. Therefore, it is difficult for a system based on the prior art to generate a correct description, while a system based on the present invention may generate a correct description because the generator is trained to generate a correct description using some signs of future events in the partial video frame.

[0060] (Interactive system based on low-latency description)

[0061] Figure 2is a block diagram showing a low-latency video description system 200 according to an embodiment of the present invention, the low-latency video description system comprising: a low-latency description generation training module 300 consisting of a training data set 301, a feature extractor 302, a future frame eliminater 303, a pre-trained description generator 320, a description generation loss calculator 309, a description similarity checker 310 and a timing detection loss calculator 311; a low-latency description module 110 consisting of a timing detector module 103 and a low-latency description generator 105; and an interaction module 201 consisting of a description understanding module 202 and a response generator 203.

[0062] The low-latency video description system 200 includes a human-machine interface (HMI) 210 that can be connected to a keyboard 211 and a pointing device / medium 212, one or more processors 220, a memory 240, a network interface controller 250 (NIC) that can be connected to a network 290 including a local area network and the Internet, a display and / or speaker interface 260 that can be connected to a display and / or speaker device 261, a machine interface 262 that can be connected to a machine actuator 263, a multimodal sensor interface 271 (such as an audio interface 273 and a visual interface 275) that can be connected to input devices including a multimodal sensor 272 (such as a microphone device 274 and a camera device 276), respectively, and a printer interface 280 that can be connected to a printing device 285. The memory 240 can be one or more memory units. The interactive system 200 based on low-latency video description can receive multimodal sensing data 295 via the network 290 connected to the NIC 250. The storage device 230 includes a feature extractor 302, a low-latency description generation training module 300, a low-latency description module 110, and an interaction module 201. In some cases, when the system is configured as a scene description system, the feature extractor 302 may be configured as a multimodal sensor feature extractor 302. In addition, when the system is configured as an audio-video description system, the feature extractor 302 may be configured as an audio-visual feature extractor 302.

[0063] To perform low-delay description, instructions may be sent to the low-delay video description system 200 using a keyboard 211, a pointing device / medium 212, or via a network 290 connected to other computers (not shown). The system 200 receives the instructions via the HMI 210 and executes the instructions using a processor 220 connected to a memory 240 by loading a low-delay description module 110 for performing low-delay video description.

[0064] The low-latency video description module 110 outputs a tag sequence as a description result for a given multimodal sensing feature sequence obtained by the feature extractor 302, and sends the tag sequence to the display / speaker device 265 via the display / speaker interface 260, to the printing device 285 via the printer interface 280, or to other computers (not shown) via the network 290. Each tag in the tag sequence can be a single word, a single letter, a single character, or a word fragment in text form.

[0065] (Training process of low-latency video description system)

[0066] Figure 3 is a block diagram showing a training module 300 included in a low-delay video description system 200 according to an embodiment of the present invention, wherein the low-delay video description system 200 utilizes a pre-trained description generator 320 to jointly optimize a low-delay description timing detector 103 and a generator 105 .

[0067] Given a training dataset 301 including a set of video clips X and a set of fact descriptions Y, where each video clip X in X is associated with its fact description Y in Y, first a feature extractor 302 converts the video clip X into a feature vector sequence Z of the video clip X. Then, a future frame remover 303 removes future frames from Z to simulate the case of low-latency video description where future frames are not available. The future frame remover 303 takes the first K feature vectors of Z, removes the rest, and generates a future-eliminated feature vector sequence Z′, where K can be randomly determined from 1≤K≤|Z| (|Z| represents the length of Z).

[0068] The low-latency description generator 105 may include an encoder 304, a decoder 305 and a search module 306, wherein the encoder 304 encodes the future elimination feature vector sequence Z′ into a hidden activation vector sequence H, and the decoder 305 estimates the random variable H in the description The posterior probability distribution on Search module 306 searches for the best description Make

[0069]

[0070] in, Instructions based on predefined vocabulary The set of all possible word sequences.

[0071] The timing detector 103 may include an encoder 307 and a timing detection module 308, wherein the encoder 307 encodes the future elimination feature vector sequence Z′ into a hidden activation vector sequence H′. The timing detector 103 may use the encoder 304 of the low-latency description generator 105 instead of the encoder 307. In this case, the timing detector 103 may not have the encoder 307, and may receive the output of the encoder 304, that is, H′ may be obtained as H′=H.

[0072] The timing detection module 308 estimates a random variable indicating whether the current timing is appropriate based on H′ The detection probability distribution on random variable It can take 0 or 1, where 0 indicates that the current timing is not appropriate, and 1 indicates that the current timing is appropriate. In the low-delay video description 100, when When F is a predetermined threshold value, the timing detector 103 can detect the appropriate timing.

[0073] To train the low-latency description generator 105, the description generation loss calculator 309 may be based on the posterior probability distribution and fact description Y to calculate the loss value. The loss value can be calculated as the cross entropy loss;

[0074]

[0075] If the encoder 304 and the decoder 305 are designed as differentiable functions such as neural networks, the parameters of the neural network can be trained using a back propagation algorithm to minimize the cross entropy loss. Minimizing the cross entropy loss means that the low-latency description generator 105 generates an appropriate description that is as close to the factual description as possible. In addition, when the system is configured as a scene description system, the encoder 304 can be configured as a multimodal sensor feature encoder 304. When the system is configured as a video description system, the encoder 304 can be configured as an audio-visual feature encoder 304.

[0076] In order to train the timing detector 103, supervisory signals are needed about whether the current timing is appropriate. However, these types of signals are not included in the training data set 301. Some embodiments of the present invention may have a description similarity checker 310 to generate a description similarity checker based on the generated description. The appropriate timing is determined by the description similarity between Z and the fact description Y. This is based on the idea that the appropriate timing is when the low-latency description generator can generate a description that is close enough to the fact description Y for the future elimination feature vector sequence Z'. Description similarity can be calculated based on any sequence similarity metric, such as word accuracy, BLEU score, and METEOR score defined between the word sequences of two descriptions.

[0077] The supervisory signal d can be calculated as

[0078]

[0079] in, yes is the similarity measure between y and Y, where S is a predefined threshold such that 0≤S<1.

[0080] Based on the supervisory signal d and the detection probability distribution estimated by the timing detection module 308 The binary cross entropy loss can be calculated as

[0081]

[0082] In addition to the low-latency description generator 105, if the encoder 307 and the timing detection module 308 are designed as differentiable functions such as neural networks, the timing detector 103 can also be trained using a back-propagation algorithm.

[0083] In some embodiments of the present invention, the training process 300 of the low-delay video description system 200 may further include a pre-trained description generator 203 to improve the performance of the low-delay video description system 200. The pre-trained description generator 203 may include an encoder 312, a decoder 313, and a search module 314, which have been trained using the training data set 301 to minimize the following loss function,

[0084]

[0085] Wherein H" can be generated by the encoder 312 without eliminating the feature vector sequence Z.

[0086]

[0087] The generated description Y' is expected to be better than the low-latency description Closer to the factual description Y, because unlike the low-latency description generator 105, the pre-trained description generator 203 can use enough video frames (without future frame elimination) to generate the description.

[0088] Similarly, the estimated posterior probability distribution Comparison Better yet, generate the correct description.

[0089] By pre-training the description generator 203, the description generation loss calculator 310 can use the Kullback-Leibler (KL) divergence loss as

[0090]

[0091] Among them, the loss value is calculated as the KL divergence indicating the similarity between two probability distributions. Minimizing the KL divergence means training the low-latency description generator 105 to mimic the pre-trained description generator 203, where the low-latency description generator 105 may use only the future-eliminated feature vector sequence Z', while the pre-trained description generator 203 may use the entire feature vector sequence Z. More specifically, the training is performed so that the low-latency description generator 105 estimates the posterior probability distribution As close as possible to the posterior probability distribution estimated by the pre-trained description generator 203 using all video frames This framework potentially avoids overfitting of the parameters of the low-latency description generator 105 to the fact description Y.

[0092] Cross Entropy Loss and KL divergence loss The linear combination of can also be used to train the low-latency description generator 105 as

[0093]

[0094] where λ is a predefined scaling factor that balances the two losses.

[0095] By pre-training the description generator 203, the description similarity checker 310 can calculate the supervision signal d as

[0096]

[0097] This mechanism encourages the timing detector 103 to detect the timing at which the low-latency description generator 105 can generate a description that is not only close to the factual description Y, but also close to the description Y' generated by the pre-trained description generator 203 for the entire feature vector sequence Z. This also avoids the supervision of the timing detector 103 from relying solely on the similarity with the factual description Y, which is expected to improve the robustness of the timing detector 103.

[0098] (Based on some implementations of audio-visual Transformer)

[0099] Figure 4 4 is a schematic diagram showing an audio-visual transformer 400 constructed as a low-delay description generator combined with a timing detector for low-delay video description according to an embodiment of the present invention. The audio-visual transformer 400 for low-delay video description includes a feature extractor 401, an encoder 402, a decoder 403 and a timing detection module 404.

[0100] Given a video stream 405, a feature extractor 401 extracts VGGish features 406 and I3D features 407 from an audio signal 408 and a visual signal 409 of an audio track and a visual track of the video stream 405, respectively, where the audio signal 408 may be an audio waveform and the visual signal 409 may be an image sequence. The frame rate of feature extraction on each track may be different. The encoder 402 encodes the VGGish 406 and I3D 407 feature vectors, where the audio and visual feature sequences from the starting point to the current time are fed to the encoder 402 and converted into a hidden activation vector sequence through a self-attention layer 410, a bimodal attention layer 411, and a feedforward layer 412. Typically, this encoder block 413 is repeated N times, for example, N=6 or greater. The final hidden activation vector sequence is obtained via the Nth encoder block.

[0101] Let X A and X V is an audio signal 408 and a visual signal 409. First, the feature extractor 401 is applied to the input signal as

[0102] A 0 =VGGish(X A ),V 0 =I3D(X V ),

[0103] To obtain feature vector sequences corresponding to VGGish 406 and I3D 407 features respectively. Each encoder block 413 calculates the hidden vector sequence as

[0104]

[0105] Among them, MHA() and FFN() represent multi-head attention and feed-forward network respectively.

[0106] MHA() takes three matrices: query Q, keyword K, and value V, each of which has its length multiplied by the feature dimension as And returns the matrix as

[0107] MHA(Q,K,V)=Concat(head1,…,head h )W O

[0108]

[0109] in, and is the parameter matrix of each multi-head attention layer, h is the number of heads, d model is the model size. Concat() concatenates matrices in the feature dimension.

[0110] The feed-forward network FFN() is calculated as

[0111] FFN(x)=max(0,xW1+b1)W2+b2,

[0112] in, and is the parameter matrix, and is the parameter vector for each feed-forward layer.

[0113] The self-attention layer 410 extracts the temporal dependencies within each modality, where the parameters of MHA() are all the same, i.e., A n-1 or V n-1 The bimodal attention layer 411 further extracts the cross-modal dependencies between the audio and visual features, where they take the keywords and values ​​from the other modality. After that, the feed-forward layer 412 is applied in a point-by-point manner. The encoded representation of the audio and visual features is obtained as A N and V N .

[0114] The timing detection module 404 receives the encoded hidden vector sequence based on the audio-visual information available at the moment. The role of the timing detection module 404 is to estimate whether the system should generate a description for a given encoded feature. The timing detection module 404 first processes the encoded vector sequence from each modality into a stacked 1D convolutional layer 414

[0115] A c =Conv1D(A N ),V c =Conv1D(V N ).

[0116] Each temporal convolution sequence is then aggregated into a single vector via pooling 415 and concatenation 416 operations.

[0117] H = Concat(MeanPool(A c ),MeanPool(V c )).

[0118] The feed-forward layer FFN() 417 and the sigmoid function σ() 418 convert the summary vector into a probability in is a random variable indicating whether the associated description can be generated.

[0119]

[0120] Once the timing detection module 404 provides a probability higher than a threshold, such as P(d=1|X A,X V )>0.5, the decoder is based on the encoded hidden vector sequence (A N ,V N ) to generate a description.

[0121] Decoder 403 starts from the starting word ( <sos>) iteratively predicts the next word. At each iteration step, the decoder receives the previously generated word 419 and estimates the posterior probability distribution of the next word 420 by applying word embedding 421, M decoder blocks 422, linear layer 423 and softmax operation 424.

[0122] Let Y i 0 Partial description after i iterations <sos>,y1,...,y i Each decoder block 422 has self-attention 425, bimodal source attention 426, concatenation 427, and feed-forward layer 428 as

[0123]

[0124] The self-attention layer 425 converts the word embedding vector into a high-level representation considering its temporal dependencies. The bimodal source attention layer 426 updates the word representation based on the correlation with the encoded multimodal hidden vector sequence. A feed-forward layer 428 is then applied to the output of the bimodal attention layer after concatenation 427. Finally, a linear transformation 423 and a softmax operation 424 are applied to the output of the Mth decoder block to obtain the posterior probability distribution of the next word 420 as

[0125]

[0126] in, Represents a vocabulary.

[0127] Choosing the best word Afterwards, the partial description is extended by adding the selected words to the previous partial description, as This is the greedy search process: if (indicates the end word), then it ends. Description The posterior probability distribution of can be calculated as

[0128]

[0129] Where H = (A N ,V N ).

[0130] Multiple words with the highest probability may also be picked and multiple description candidates may be considered in the search module 306 according to a beam search technique.

[0131] (train)

[0132] The multimodal encoder, timing detector, and description decoder are jointly trained so that the model can achieve description quality comparable to the full video even when the given video is shorter than the original (by truncating the second half).

[0133] For the model = (θ C ,θ D ) The training process repeats the following steps:

[0134] To T o ~Uniform(T s ,T e )sampling,

[0135] Calculating Losses use Update θ

[0136] (reasoning)

[0137] This reasoning is performed in two steps:

[0138] Find first satisfaction t,

[0139] based on

[0140] Generate a description.

[0141] (Evaluating low-latency video description quality)

[0142] Figure 5 is an evaluation result obtained by executing the video description benchmark according to an embodiment of the present invention.

[0143] The proposed low-latency description generation is tested using the ActivityNet Description Dataset (Krishna et al. (2017)), which consists of 100,000 description sentences associated with temporal localization information based on 20,000 YouTube videos. The dataset is split into 50%, 25%, and 25% for training, validation, and testing. The validation set is split into two subsets, and the performance is reported on these two subsets. The average duration of video clips is 35.5 seconds, 37.7 seconds, and 40.2 seconds for the training set and validation subsets 1 and 2, respectively. VGGish features and I3D features are used. The VGGish features are configured to form a 128-dimensional vector sequence for the audio track of each video, where each audio frame corresponds to a 0.96-second segment with no overlap. The I3D features are configured to form a 1024-dimensional vector sequence for the video track, where each visual frame corresponds to a 2.56-second segment with no overlap.

[0144] First, a multimodal Transformer was trained using all video clips and their factual descriptions. This model was used as a baseline and teacher model. N=2 for the encoder block, M=2 for the decoder block, and the number of attention heads was 4. The vocabulary size was 10172 and the dimension of the word embedding vector was 300.

[0145] The proposed online description model was trained using incomplete video clips according to the steps in the present invention. During training, α=β=γ=1 / 3 was used for the loss function. The dimensions of the hidden activations in the audio and visual attention layers were 128 and 1024, respectively. The timing detector had 2 stacked 1D convolutional layers with ReLU nonlinearity between them. The performance was measured by BLEU3, BLEU4, and METEOR scores.

[0146] The delay ratio indicates the ratio of the video duration used for description to the duration of the original video clip. For the baseline model, the delay ratio is always 1, which means that all frames are used to generate descriptions.

[0147] Figure 5 The methods are compared using the METEOR score on validation subset 1. The model selected for evaluation was trained with S = 0.6 (determined empirically) and had the best METEOR score on validation subset 2. The delay controlling the detection threshold F was Figure 5 As shown, at 55% latency the proposed method achieves a 10.45 METEOR score with only a small degradation, which corresponds to 98% of the baseline score of 10.67. It also achieves a 10.00 METEOR score at 28% latency, which corresponds to 94% of the baseline. An original method was tested that starts by taking video frames at a fixed ratio to the original video length and runs the baseline description on the truncated video clips. The results show that at equivalent latency the proposed method significantly outperforms the original method.

[0148] The table also includes the results of using a unimodal Transformer that receives only visual features. This result shows that the proposed method works well for only visual features, but the performance drops due to the lack of audio features. This result shows that audio features are crucial even in the proposed low-latency method.

[0149] The low-latency description is input into an interactive system trained using a pair of description and action commands to understand the scene and decide the next action. The interactive system can use a sequence-to-sequence neural network model that converts the received description into a sequence of words for responding to humans or a sequence of action commands for controlling a robot.

[0150] In addition, some embodiments are based on the following recognition: scene-aware interaction technology allows machines to interact with humans based on shared knowledge obtained by using various sensors to identify and understand their surroundings. Audio-visual scene-aware dialogue (AVSD) is one of the scene-aware interaction technologies. According to some recognitions based on AVSD, end-to-end methods can better handle flexible conversations between users and systems by training models on large conversation data sets. These methods have been extended to perform conversations about objects and events occurring around machines or users based on the understanding of dynamic scenes captured by multimodal sensors such as cameras and microphones. The extension allows users to ask questions about what is happening in their surroundings. For example, the framework is applicable to visual question answering (VQA) for one-time QA of static images, visual dialogue, video QA for one-time QA of video clips, and AVSD, in which the AI ​​agent uses natural conversation language to maintain a meaningful conversation with humans about static images, and the AVSD performs a QA-based conversation with humans about video clips. Video QA is a QA (question-answering) task that consists of answering a single question about a video clip such as found on YouTube, and is formalized in the MSVD (Microsoft Research Video Description Corpus) and MSRVTT (Microsoft Research Video to Text) datasets. AVSD is a multi-turn dialogue task that consists of generating responses to user questions about video clips of everyday life, and has been solved in the Dialog Systems Technology Challenges (DSTC). AVSD systems are trained to generate sentences to answer user questions about video clips, where the system needs to understand what events happened, when events happened, how events happened, and who the events happened to based on the temporal sequence of audio-visual features to provide the correct answer.

[0151] At any moment, countless events happening in the real world are captured by cameras and stored as massive video data resources. To efficiently retrieve such recordings, both in offline and online settings, video description has been actively studied in the field of computer vision since the S2VT (Sequence-to-Sequence - Video-to-Text) system was first proposed, adopting an end-to-end sequence-to-sequence model. Its goal is to generate video descriptions (descriptions) about objects and events in video clips. To further leverage audio features to identify events, multimodal attention methods allow the fusion of audio and visual features, such as VGGish (audio classification model similar to the Visual Geometry Group (VGG)) and I3D (Interactive 3D) to generate video descriptions. This video clip description technique has been extended to offline video stream description techniques, such as Dense Video Description and Line-by-Line Video Description Generator, where all salient events in a video stream are temporally localized and event-triggered descriptions are generated in a multi-threaded manner. While all video description techniques to date are based on LSTM, Transformer can be successfully applied in conjunction with the audio-visual attention framework.

[0152] Current video QA dialogue systems usually process offline video data and generate answers to questions after the clip is finished. In order for interactive systems to achieve real-time dialogue about scenes for surveillance and robotics applications, they need to understand scenes and events and respond quickly to user queries about scenes. In this work, the audio-visual Transformer is tested using the ActivityNet description dataset within an offline video description system and achieves state-of-the-art performance on dense video description tasks.

[0153] However, this scene-aware interaction task assumes that the system has access to all frames of the video clip when predicting the answer to the question. This is unrealistic for a real-time conversation system that is monitoring an ongoing audio-visual scene, because the system not only needs to accurately predict the answer, but also needs to quickly find question-related events in the online video stream and generate appropriate answers to respond to the user as quickly as possible. This capability requires the development of new low-latency QA techniques.

[0154] In previous work, we proposed a low-latency audio-visual description method that can accurately and quickly describe events without waiting for the end of a video clip, optimizing the description generation time. At the same time, we recently introduced a new AVSD task in the 3rd AVSD Challenge of DSTC10, specifically requiring the system to demonstrate temporal reasoning capabilities by finding evidence from the video to support its answer. This is based on a new extension of the AVSD dataset of DSTC10, for which we collected human-generated temporal reasoning data. This work proposes to extend our low-latency description method to scene-aware interaction tasks, combining it with our reasoning AVSD system to develop a low-latency online scene-aware interaction system.

[0155] Some extensions to the low-latency video description work allow for a novel approach that optimizes the time to generate each answer in a trade-off between generation latency and answer quality. For video QA, the timing detector is now responsible for finding the timing of question-relevant events, rather than determining when the system has seen enough information to generate a general description as in the video description case. The audiovisual scene-aware dialogue system built for the DSTC10 Dialogue Systems Technical Challenge can be extended to exploit low-latency features. For example, experiments using the MSRVTT-QA and AVSD datasets show that our approach achieves 97% to 99% of the upper bound on the quality of answers given by a pre-trained Transformer using the entire video clip, starting with less than 40% of the first frames.

[0156] In the same idea as the low-latency description method, another system according to another embodiment optimizes the output timing of each answer based on the trade-off between latency and answer quality. A low-latency audio-visual Transformer is trained, which consists of the following parts: (1) a Transformer-based response generator that attempts to generate factual answers after seeing only a small portion of all video frames and also simulates the output of a similar pre-trained response generator that is allowed to see the entire video, and (2) a CNN-based timing detector that can find the optimal timing to output answers for the input questions so that the responses finally generated by the above two Transformers become close enough to each other. Once the relevant event occurs, the proposed jointly trained response generator and timing detector can generate answer responses at the early stage of the video clip and even predict future frames. Since information from multiple modalities is combined, the system has more opportunities to rely on the earliest clues in a certain modality to recognize events early. Experiments using MSR-VTT QA and AVSD datasets show that our method achieves low-latency video QA with competitive answer quality compared to offline video QA baselines that use the entire video frames.

[0157] According to one embodiment of the present invention, a video QA is provided, which is different from video description, to generate appropriate answers to user questions as quickly as possible. To this end, a question encoder is introduced to provide question embeddings to the timing detector, and the sentence generator is extended to accept questions as context information. Therefore, while the proposed method utilizes the same underlying mechanism for low-latency processing, the model is extended to be used for video QA tasks.

[0158] (Low-latency video QA model)

[0159] We build our proposed model for low-latency video QA on top of DSTC10-AVSD, which adopts the AV-transformer architecture. For the DSTC10-AVSD challenge, we extend the AV-transformer by introducing joint teacher-student learning and attention multimodal fusion, achieving state-of-the-art performance. As in our low-latency video description system, the low-latency QA model receives video and audio features in a streaming manner, and a timing detector decides when to generate a response for the sequence of features that the model has received up to that moment. Figure 6 A model architecture (low-latency video QA architecture) 600 for low-latency video QA according to some embodiments of the present invention is shown. The model architecture 600 includes parts similar to the functional parts used in the audio-visual Transformer 400. The same numbers as the parts (or layers) in the audio-visual Transformer 400 are used in the figure. The low-latency video QA architecture 600 also includes word embeddings 621, a question encoder (text encoder) 613 with a self-attention layer 610 and a feed-forward layer 612, a pooling 615 connected to the splicing 416, the AV encoder 402, the timing detector 404 and the response decoder 403, wherein the AV encoder 402 is shared by the timing detector 404 and the response decoder 403.

[0160] Furthermore, the mathematical expressions or equations used to explain the audio-visual Transformer 400 are used in the following description to explain the low-latency video QA architecture 600 .

[0161] Given a video stream and question text as input, the AV encoder encodes the VGGish and I3D features extracted from the audio and video tracks, while the Transformer-based text encoder encodes the question. The audio and visual feature sequences from the starting point to the current time are fed to the encoder and converted into hidden vector sequences through self-attention, bimodal attention, and feedforward layers. This encoder block is repeated N times, and the final encoded representation is obtained via the Nth encoder block. This encoder block is repeated multiple times, and the final encoded representation is obtained via the Nth encoder block. The question word sequence is also encoded via a word embedding layer and then processed by a Transformer with N′ blocks.

[0162] We use X A and X V Denote the audio and visual signals, and Q denotes the question sentence. First, the feature extractor 401 is applied to the input signal as

[0163] A 0 =VGGish(X A ),V 0 =I3D(X V ), (1)

[0164] To obtain feature vector sequences corresponding to VGGish and I3D features respectively. Each encoder block calculates the hidden vector sequence as

[0165]

[0166] Where MHA and FFN denote multi-head attention and feed-forward network, respectively. Layer normalization is applied before each MHA and FFN layer, but is omitted from the equation for simplicity. MHA takes three independent variables: query, key, and value vector sequences. The self-attention layer extracts temporal dependencies within each modality, where the independent variables of MHA are all the same, i.e., A n-1 or V n-1 , as in (2) and (3). The bimodal attention layer further extracts the cross-modal dependencies between audio and visual features, obtaining keys and values ​​from the other modality as in (4) and (5). The feed-forward layer is then applied in a point-by-point manner. The encoded representation A of the audio and visual features is obtained. N and V N Question Q is also encoded using a Transformer with N′ blocks as

[0167] Q 0 =WordEmbedding(Q),(8)

[0168]

[0169] Using the AV encoder and the question encoder, we obtain the encoding representations A and V of the audio-visual features and the question word sequence The timing detector receives the encoded representations A and V available as of the current time and the encoded question Q. The role of the timing detector is to decide whether the system should generate an answer for a given encoded feature and question. The detector first processes the sequence of encoded vectors from each modality with stacked 1D convolutional layers (Conv1d) as

[0170] A′=Conv1d(A), V′=Conv1d(V). (11)

[0171] These temporal convolution sequences and the question encoding are then summarized into a single vector via average pooling and concatenation operations:

[0172]

[0173] The feed-forward layer FFN and the sigmoid function σ convert the summary vector into a probability of d, where d represents whether a relevant answer can be generated:

[0174]

[0175] Once the timing detector output probability is above a threshold (eg, P(d=1|A,V,Q)>0.5), the decoder generates a reply based on the encoded representation.

[0176] The decoder starts from the question word sequence Add the start tag <sos>It starts by iterating and predicting the next word. In each iteration step, it receives the generated partial answer sentence and predicts the next word by applying M decoder blocks and prediction network. Assume Y i is the partial answer statement after i iterations <sos>,y1,…,y i ,and is the sequence of problem conditions Each word in the question and partial answer sentences is converted into a word embedding vector. Each decoder block has self-attention, bimodal attention and feed-forward layers:

[0177]

[0178]

[0179] Where MHA stands for multi-head attention network. In (14), the self-attention layer maps the word embedding vectors to a high-level representation that contains their temporal dependencies. The bimodal attention layer updates the word representation based on the correlation with the encoded bimodal representation in (15) and (16). In (17) and (18), the feed-forward layer transforms the concatenated output of the bimodal attention layer. These operations are repeated to the Mth block. A linear transformation and a softmax operation are applied to the output of the Mth block to obtain the posterior probability distribution of the next word as

[0180]

[0181] in, Represents a vocabulary. Some answers are given by adding the best words Append to the previous part of the statement to expand it, that is: This is a greedy search process that repeats until the end-of-sentence token is received. <eos>As A beam search technique can also be used, which selects the top K words with the highest probability and keeps multiple word sequences. Finally, the best word sequence is selected as the final answer to the input question.

[0182] (Training and Inference)

[0183] The AV encoder, question encoder, response decoder, and timing detector are jointly trained so that the system can achieve answer quality comparable to the full video even when the given video is shorter than the original (by truncating the second half).

[0184] Two loss functions are used, a response loss to improve the answer quality, and a timing detection loss to detect the appropriate timing of issuing the answer sentence. Figure 7 shows a video stream, where the video has been s Start and answer Y with the facts e ′ associated with T e End. If you choose time T o As the timing of issuing, the response decoder is based on the time T s To time T o Audio-visual signal and Problems to generate answers.

[0185] Response loss is based on the fact that answer Y e′ The standard cross entropy loss is

[0186]

[0187] and the Kullback-Leibler (KL) divergence loss between the predictions of the pre-trained model that is allowed to process complete videos and the predictions of the target model that can only process incomplete videos, i.e.,

[0188]

[0189] This teacher-student learning method can utilize the teacher model The model uses the entire video clip to Predict the answer by pushing the student model θ R , using only truncated video clips to mimic the teacher’s predictions. This improves training stability and leads to better performance.

[0190] The timing detection loss is based on the binary cross entropy of proper timing. However, typically, such timing information is not present in the training dataset. As described in [? ], the response decoder is based on whether it can generate a proper answer (i.e., close enough to the factual answer Y e′ Or use a pre-trained model For the entire video clip Generated answers ) to determine the correct timing. The detection loss is calculated as

[0191]

[0192] Where d is determined based on the following formula:

[0193]

[0194] where Sim(·,·) represents the similarity measure between two word sequences. In this work, the percentage of matching words calculated in a forced teaching manner is used, where and As the sequence of the highest probability words, given the fact word sequence as the left context, and count the matching words between them. S∈(0,1] is the judgment of the online answer Is it close enough to the reference Y e′ and a predetermined threshold.

[0195] By repeating the timing T o ~Uniform(T s ,T e ) to sample and calculate the loss And use The step of updating the parameters θ to train the model θ = (θ R ,θ D ).

[0196] At inference time, the emission time Determined to be the first to meet time, where F is a predetermined threshold that controls the sensitivity of the timing detection. The answer is then generated based on the following formula:

[0197]

[0198] Note that, assuming T s It has been confirmed.

[0199] According to some embodiments of the present invention, an artificial intelligence (AI) low-latency processing system is provided. Figure 8 A computer-implemented method 800 is shown including processing steps performed by a low-latency processing system using a processor and memory or memory storage according to one embodiment of the present invention.

[0200] The low-latency processing system may include: a processor; and a memory, in which instructions are stored as a computer-implemented method 800, which, when executed by the processor, causes the low-latency processing system to perform the following operations: collect a frame sequence, the frame sequence jointly including information dispersed in at least some frames in the frame sequence; execute a timing neural network, which is trained to identify an early frame subsequence in the frame sequence, the early frame subsequence including at least partial information indicating the information; execute a decoding neural network, which is trained to decode the information from partial information in the frame subsequence, wherein the timing neural network and the decoding neural network are jointly trained to iteratively identify a minimum number of subframes starting from the beginning of the training frame sequence, the subframes containing partial training information sufficient to decode the training information.

[0201] In some cases, the timing detector neural network is jointly trained with the decoder neural network on features of different subsequences of the frame sequence to minimize a multi-task loss function including a temporal detection loss and an information generation loss. The multi-task loss function may include three losses that define (1) the accuracy of the decoded information, (2) the difference between the information decoded from the frame subsequence and the information decoded from the complete frame sequence, and (3) the accuracy of the prediction of the timing detector neural network. In addition, the processor can be configured to execute a feature extractor neural network to extract features from each frame in the frame sequence; execute a feature encoder neural network to encode the extracted features of each frame to produce an encoded feature sequence; submit the encoded feature sequence to the timing detector neural network and identify the encoded feature subsequence representing the frame subsequence; and submit the encoded feature subsequence to the decoder neural network to decode the information. For other cases, the processor can trigger the execution of the module of the AI ​​low-latency processing system when a new input frame is received that is appended to the frame sequence. The information can be a description of an audio scene, a video scene, or an audio-video scene. In addition, the information is an answer to a question about the frame sequence. In some cases, the frame may include multi-model information from different sensors of different modalities. Additionally, the information may be an answer to a question about the sequence of frames, wherein the processor executes a text encoder neural network to encode the question, submits the encoded question to the timing neural network, and submits the question or the encoded question to the decoding neural network.

[0202] (experiment)

[0203] The MSRVTT-QA and AVSD datasets are used to evaluate our low-latency video QA method. MSRVTT-QA is based on the MSR-VTT dataset, which contains 10k video clips and 243k question-answer (QA) pairs. QA pairs are automatically generated from the manually annotated descriptions of each video clip, where questions are sentences and answers are single words. We follow the data split in the MSR-VTT dataset, where 65% is used for training, 5% is used for validation, and 30% is used for testing. AVSD is a set of text-based dialogues on short videos from the Charades dataset, which consists of untrimmed multi-action videos, each of which includes an audio track. In AVSD, two parties (called questioners and responders) have a dialogue about events in the provided video. The job of the responder, who has watched the video, is to answer the questions asked by the questioner. We follow the AVSD challenge setting, where the training, validation, and test sets consist of 7.7k, 1.8k, and 1.8k dialogues, respectively, and each dialogue includes 10 rounds of QA pairs, where both questions and answers are sentences. The duration of the video clips ranged from 10 seconds to 40 seconds.

[0204] The VGGish features are configured to form a sequence of 128-dimensional vectors for the audio track of each video, where each audio frame corresponds to a 0.96-second segment with no overlap. The I3D features are configured to form a sequence of 2048-dimensional vectors for the video track, where each visual frame corresponds to a 2.56-second segment with no overlap.

[0205] We first trained a multimodal Transformer with the entire video clip and QA pairs. This model is used as the baseline and teacher model. We use N = 2 audio-visual encoder blocks, N′ = 4 question encoder blocks, M = 4 decoder blocks, and set the number of attention heads to 4. The vocabulary size is 7599 for MSRVTT-QA and 3669 for AVSD. The dimension of the word embedding vector is 300.

[0206] The proposed model for low-latency video QA is trained with incomplete video clips according to the steps in Section 3.2. The architecture is the same as the baseline / teacher model except for the addition of the timing detector. During training, we always use α = β = γ = 1 / 3 for the loss function and threshold S = 0.9 in Eq. (24). We set the dimensions of the hidden activations in the audio and visual attention layers to 256 and 1024, respectively, set the dropout rate to 0.1, and apply a label smoothing technique. The timing detector consists of 2 stacked 1D convolutional layers with a ReLU nonlinearity between them. Performance is measured by answer accuracy on MSRVTT-QA and BLEU4 and METEOR scores on AVSD.

[0207] Fig. 9 The relationship between latency ratio and answer accuracy on MSRVTT-QA is shown. The latency ratio represents the ratio of frames actually used (from the start) to all video frames. The baseline results are obtained using the baseline (teacher) model by simply omitting various ratios of future frames. The results of the proposed model are obtained by changing the detection threshold F. The accuracy of MSRVTT-QA is expressed as the percentage of answers to single words that are consistent with the facts. The results show that our proposed method achieves low-latency video QA with much smaller accuracy degradation compared to the baseline. Our method achieves 97% of the upper limit answer quality given by the pre-trained Transformer using the entire video clip using only 40% of the frames from the start.

[0208] Fig.10 Figure 2 shows a comparison of the quality of answer sentences in the AVSD task. We control the delay ratio by setting the detection threshold F to obtain average ratios of 1.0, 0.5, and 0.2 for the proposed method. For the baseline system, future frames with the above fixed ratio are omitted. As shown in the table, our proposed method slightly outperforms the baseline even with a delay of 1.0. This may be due to the improved robustness of the model by training with randomly truncated videos. In addition, the proposed method maintains the same level of BLEU4 and METEOR scores at 0.5 delay, and still achieves competitive scores even at 0.2 delay, with only a slight degradation, reaching 98% to 99% of the scores under the 1.0 delay condition.

[0209] Fig.11A and Fig. 11B The QA distributions under the delays set with detection thresholds F = 0.3 and F = 0.4 in the AVSD-DSTC7 task are shown, which correspond to Fig.10 Results for delays of 0.2 (left) and 0.5 (right) in . These results suggest that most QAs for AVSD require only early frames or all frames to generate accurate responses. We investigate the reasons for the polarized distribution. The most common pattern for questions leading to early decisions is “How does the video start?”. Furthermore, in the training data, there are some consistent responses, such as answering “One person” in response to “How many people are in the video?”. This common language pattern may also be a reason for early decisions. Cases for late decisions contain patterns such as “How does the video end?”. This type of question is natural for a questioner who needs to generate a video description through 10 QAs without watching the full video.

[0210] According to some of the above embodiments, the low-latency video QA method can accurately and quickly answer the user's questions without waiting for the end of the video clip. The proposed method optimizes the output timing of each answer based on the trade-off between latency and answer quality. When using the MSRVTT-QA and AVSD datasets, the above system can generate answers at the early stage of the video clip, using less than 40% of the frames from the beginning to achieve 97% to 99% of the upper limit answer quality given by the pre-trained Transformer using the entire video clip.< / eos> < / sos> < / sos> < / sos> < / sos>

Claims

1. An artificial intelligence (AI) low-latency processing system, the AI ​​low-latency processing system comprising: processor; and a memory storing instructions that, when executed by the processor, cause the AI ​​low-latency processing system to: collecting a sequence of frames, the sequence of frames jointly comprising information dispersed in at least some of the frames in the sequence of frames; executing a timing neural network trained to identify an early subsequence of frames in the sequence of frames that includes information indicative of at least a portion of the information; executing a decoding neural network trained to decode the information from the portion of information in the frame subsequence, The timing neural network and the decoding neural network are jointly trained to iteratively identify a minimum number of subframes starting from the beginning of a training frame sequence that contain partial training information sufficient to decode the training information.

2. The AI ​​low-latency processing system according to claim 1, wherein: The timing detector neural network and the decoder neural network are jointly trained on features of different subsequences of the frame sequence to minimize a multi-task loss function including a temporal detection loss and an information generation loss.

3. The AI ​​low-latency processing system according to claim 2, wherein: The multi-task loss function includes three losses, which define: (1) the accuracy of the decoded information; (2) the difference between the information decoded from the frame subsequence and the information decoded from the complete frame sequence; and (3) the prediction accuracy of the timing detector neural network.

4. The AI ​​low-latency processing system according to claim 1, wherein: The processor is configured to: executing a feature extractor neural network to extract features from each frame in the sequence of frames; executing a feature encoder neural network to encode the extracted features of each frame to produce an encoded feature sequence; submitting the coded signature sequence to the timing detector neural network and identifying a coded signature subsequence representing the frame subsequence; as well as The encoded feature subsequence is submitted to the decoder neural network to decode the information.

5. The AI ​​low-latency processing system according to claim 1, wherein: The processor triggers execution of a module of the AI ​​low-latency processing system upon receiving a new input frame appended to the frame sequence.

6. The AI ​​low-latency processing system according to claim 1, wherein: The information is a description of an audio scene, a video scene or an audio-visual scene.

7. The AI ​​low-latency processing system according to claim 1, wherein: The information is answers to questions about the sequence of frames.

8. The AI ​​low-latency processing system according to claim 4, wherein: The information is an answer to a question about the sequence of frames, wherein the processor is configured to: executing a text encoder neural network to encode the question; submitting the encoded question to the timed neural network; and The question or the encoded question is submitted to the decoding neural network.

9. The AI ​​low-latency processing system according to claim 1, wherein: The frame includes multi-model information from different sensors of different modalities.

10. A computer-implemented method for an artificial intelligence (AI) low-latency processing system, the AI ​​low-latency processing system comprising a processor and a memory, the memory storing instructions for using the processor to perform the following steps of the computer-implemented method, the steps comprising: collecting a sequence of frames, the sequence of frames jointly comprising information dispersed in at least some of the frames in the sequence of frames; executing a timing neural network trained to identify an early subsequence of frames in the sequence of frames that includes information indicative of at least a portion of the information; as well as executing a decoding neural network that is trained to decode the information from the portion of information in the subsequence of frames, wherein the timing neural network is jointly trained with the decoding neural network to iteratively identify a minimum number of subframes starting from the beginning of a sequence of training frames that contain sufficient portion of training information to decode the training information.

11. The computer-implemented method of claim 10, wherein: The timing detector neural network and the decoder neural network are jointly trained on features of different subsequences of the frame sequence to minimize a multi-task loss function including a temporal detection loss and an information generation loss.

12. The computer-implemented method of claim 11, wherein: The multi-task loss function includes three losses, which define: (1) the accuracy of the decoded information; (2) the difference between the information decoded from the frame subsequence and the information decoded from the complete frame sequence; and (3) the prediction accuracy of the timing detector neural network.

13. The computer-implemented method of claim 10, wherein: The processor is configured to: executing a feature extractor neural network to extract features from each frame in the sequence of frames; executing a feature encoder neural network to encode the extracted features of each frame to produce an encoded feature sequence; submitting the coded signature sequence to the timing detector neural network and identifying a coded signature subsequence representing the frame subsequence; as well as The encoded feature subsequence is submitted to the decoder neural network to decode the information.

14. The computer-implemented method of claim 10, wherein: The processor triggers execution of a module of the AI ​​low-latency processing system upon receiving a new input frame appended to the frame sequence.

15. The computer-implemented method of claim 10, wherein: The information is a description of an audio scene, a video scene or an audio-visual scene.

16. The computer-implemented method of claim 10, wherein: The information is answers to questions about the sequence of frames.

17. The computer-implemented method of claim 10, wherein: The frame includes multi-model information from different sensors of different modalities.

18. The computer-implemented method of claim 13, wherein: The information is an answer to a question about the sequence of frames, wherein the processor is configured to: executing a text encoder neural network to encode the question; submitting the encoded question to the timed neural network; and The question or the encoded question is submitted to the decoding neural network.

Images