Low-latency captioning system

Abstract

An artificial intelligence (AI) low-latency processing system is provided. The low-latency processing system includes a processor and a memory storing instructions. The low-latency processing system is configured to collect a sequence of frames, the sequence of frames jointly including information distributed among at least some of the frames of the sequence of frames. The system is further configured to execute a timing neural network trained to identify a subsequence of an initial frame of the sequence of frames that includes at least a portion of the information indicating the information, and to execute a decoding neural network trained to decode information from a portion of the information of the subsequence of frames, and the timing neural network is co-trained with the decoding neural network to repeatedly identify a minimum number of sub-frames from the beginning of a training sequence of frames that includes a portion of the training information sufficient to decode the training information.

Description

[Technical Field]

[0001] (Cross-reference of related applications) This application is a continuation-in-part application of Patent Application No. 17 / 384,234, filed on 23 July 2021, the contents of which are incorporated herein by reference.

[0002] The present invention generally relates to scene captioning and interaction based on scene captioning, and more specifically to end-to-end scene captioning that utilizes latency optimization capabilities. [Background technology]

[0003] At any given moment, countless events occurring in the real world are captured by sensors such as cameras, microphones, LiDAR, and mmWave, and stored as a vast amount of sensor data resources. To effectively acquire such records, scene captioning is an essential technology, whether in an offline or online setting, as it has the ability to understand scenes and describe events in natural language. Furthermore, robots can interact with humans or other robots to determine their next actions based on their understanding of the scene through online scene captioning.

[0004] Scene captioning techniques have been actively researched in the field of computer vision for video captioning. Deep recurrent neural networks (RNNs) have been applied to video captioning, and RNNs are trained to convert sequences of image features extracted from video clips into sequences of words that describe the video content. The goal is to generate video descriptions (captions) for objects and events within any given video clip. Transformer models have become more popular than RNNs as video captioning models in recent years because they offer improved captioning performance.

[0005] Furthermore, not only image features but also audio features are extracted from video clips and used to improve caption quality, and attention-based multimodal fusion technology has been introduced to effectively fuse image and audio features according to the video content (U.S. Patent No. 10,417,498).

[0006] Traditional methods for video captioning are assumed to function primarily offline, where each video clip is provided before captioning, allowing the system to access all frames of the video clip to generate captions. However, such traditional methods are impractical for real-time monitoring and supervision systems, car navigation, and scene-understanding-based dialogue systems for robots. This is because, in such cases, it is essential not only to accurately describe events but also to generate captions as quickly as possible to quickly detect and report events and take the next action. Therefore, low-latency captioning for online systems is needed to achieve this functionality. The system needs to determine the appropriate timing to generate the correct caption using only a limited number of frames it has received up to that point. In addition, such low-latency captioning capabilities enable robots to determine the next action as quickly as possible in order to interact with humans and other robots. [Overview of the Initiative]

[0007] Some embodiments of this disclosure are based on the recognition that scene captioning is an essential technique for understanding scenes and describing events in natural language. To apply this to real-time monitoring systems, the system needs to not only accurately describe events but also generate captions as quickly as possible. Achieving such functionality requires low-latency captioning, but the research field of such online scene captioning has not yet been achieved. This specification proposes a novel method for optimizing the output timing of each caption based on a balance between latency and caption quality. An audio-visual transformer is trained to generate ground-truth captions using only a small portion of the entire scene, and all scenes are trained to mimic the output of a given pre-trained transformer. A CNN-based timing detector is also trained to detect appropriate output timings, in which case the captions generated by the two transformers will be sufficiently close to each other. Using the jointly trained transformers and timing detector, captions can be generated immediately when an event occurs, or in the early stages of an event if the event is predictable.

[0008] Objectives of some embodiments of the present invention are to provide systems and methods for end-to-end scene captioning that enable online / offline supervision systems and scene-aware interaction (dialogue) to understand events using natural language as soon as possible once the system recognizes an event.

[0009] This disclosure includes a low-latency scene captioning system that is trained to optimize output timing for each caption based on a trade-off between latency and caption quality. The present invention may train a low-latency caption generator according to the following strategies: (1) Generate ground-true captions using a low-latency caption generator that recognizes only a small portion of the entire scene or a small portion of all signals acquired by the sensor. (2) Mimic the output of a pre-trained caption generator, which is generated using the entire scene. (3) Train a timing detector to find the best timing to output captions so that the captions ultimately produced by the low-latency caption generator are sufficiently close to the ground-true captions or sufficiently close to the captions generated by the pre-trained caption generator using the entire scene. The low-latency caption generator based on (1) and (2) and the timing detector in (3) are trained together.

[0010] A jointly trained low-latency caption generator and timing detector can generate captions immediately upon event occurrence, at an early stage of the scene captured by the sensor. In addition, this framework can be applied to predict future events during low-latency captioning. Furthermore, by combining multimodal detection information, events can be recognized at an earlier timing, triggered by the earliest cue in one of the modalities, without waiting for other cues in other modalities. In particular, by using an audio-visual transformer constructed as a low-latency caption generator according to an embodiment of the present invention, captions can be generated based on the timing of audio cues, earlier than the timing of visual cues. Such a low-latency video captioning system using multimodal detection information can contribute not only to rapid event retrieval but also to faster response to scenes.

[0011] Some embodiments are based on the understanding that experiments using the ActivityNet Captions dataset have shown that systems based on the present invention can achieve 94% of the upper limit of caption quality given by a pre-trained transformer that uses the entire video clip, by using only the first 28% of the frame.

[0012] According to some embodiments of the present invention, a scene captioning system can be provided. In this case, the scene captioning system includes an interface configured to acquire a stream of signals captured by a multimodal sensor for captioning a scene, and a memory for storing a computer-executable scene captioning model including a multimodal sensor feature extractor, a multimodal sensor feature encoder, a timing detector, and a scene caption decoder, wherein the multimodal sensor feature encoder is shared by the timing detector and the scene caption decoder, and the scene captioning system further includes a processor associated with the memory, the processor configured to perform the steps of: extracting multimodal sensor features from a multimodal sensor signal using the multimodal sensor feature extractor; encoding the multimodal sensor features using the multimodal sensor feature encoder; and determining the timing for generating a scene caption using the timing detector, the timing being set at an early stage in the stream of the multimodal sensor signal, and the processor further configured to perform the step of generating a scene caption describing an event based on the multimodal sensor features using the scene caption decoder according to the timing.

[0013] Furthermore, several embodiments of the present invention are based on the recognition that a computer-executable training method is provided for training a multimodal sensor feature encoder, a timing detector, and a scene caption decoder. The method may include the steps of: providing a training dataset including a set of multimodal sensor signals and a set of ground truth scene captions; converting the multimodal sensor signals into a feature vector sequence using a feature extractor; and removing future frames from the feature vector sequence using a future frame excluder, the future frame excluder taking a first feature vector from the feature vector sequence and removing other feature vectors to generate a future-excluded feature vector sequence; further including the steps of: encoding the future-excluded feature vector sequence into a hidden activation vector sequence; training a low-latency scene caption generator by calculating a loss value; calculating the loss value based on a posterior probability distribution and ground truth captions; and training a timing detector by calculating a supervisor signal based on the scene caption similarity between the scene captions for the future-excluded feature vector sequence and the ground truth captions.

[0014] Furthermore, if the scene captioning system is configured as a video captioning system, the video captioning system may include an interface configured to acquire a stream of audio-visual signals including image data and audio data, and memory for storing a computer-executable video captioning model including an audio-visual feature extractor, an audio-visual feature encoder, a timing detector, and a video caption decoder. The audio-visual encoder is shared by the timing decoder, the timing detector, and the video caption decoder, and the video captioning system may further include a processor associated with the memory. The processor is configured to perform the steps of: extracting audio and visual features from an audio-visual signal using an audio-visual extractor; encoding the audio and visual features from the audio-visual signal using an audio-visual encoder; and determining the timing for generating video captions using a timing detector, the timing being set at an early stage of the stream audio-visual signal. The processor is further configured to perform the step of generating video captions describing an audio-visual scene based on the audio and visual features using a video caption decoder in accordance with the timing.

[0015] According to several embodiments of the present invention, an artificial intelligence (AI) low-latency processing system is provided. The low-latency processing system may include a processor and a memory storing instructions. When an instruction is executed by the processor, it causes the low-latency processing system to collect a sequence of frames, the sequence of frames jointly containing information distributed across at least some of the frames in the sequence of frames. The instruction further causes a timing neural network to execute, which is trained to identify a subsequence of early frames in the sequence of frames that contains at least a portion of the information indicating the information, and a decoding neural network to execute, which is trained to decode information from the portion of the information in the subsequence of frames. The timing neural network is trained jointly with the decoding neural network to repeatedly identify a minimum number of subframes from the beginning of the training sequence of frames that contain a portion of the training information sufficient to decode the training information.

[0016] Furthermore, one embodiment of the present invention provides a computer-implemented method for an artificial intelligence (AI) low-latency processing system. The low-latency processing system includes a processor and a memory storing instructions for a computer-implemented method that uses the processor to perform steps. The computer-implemented method includes the step of collecting a sequence of frames. The sequence of frames jointly includes information distributed across at least some of the frames in the sequence of frames. Furthermore, it includes the step of running a timing neural network trained to identify a subsequence of early frames in the sequence of frames that includes at least a portion of the information indicating the information, and the step of running a decoding neural network trained to decode information from the portion of the information in the subsequence of frames. The timing neural network is trained jointly with the decoding neural network to repeatedly identify a minimum number of subframes from the beginning of the training sequence of frames that include a portion of the training information sufficient to decode the training information. [Brief explanation of the drawing]

[0017] Embodiments of the present disclosure will be further described with reference to the accompanying drawings. The drawings shown are not necessarily to scale and are generally intended to illustrate the principles of the embodiments of the present disclosure.

[0018] [Figure 1] This is a schematic diagram illustrating a low-latency video captioning process according to an embodiment of the present invention. [Figure 2] Block diagram showing a low-latency video captioning system according to an embodiment of the present invention, including a low-latency caption generator, a pre-trained caption generator, and a timing detector. [Figure 3]A block diagram showing a training procedure for a low-latency caption generator and a timing detector that jointly optimize a pre-trained caption generator to generate low-latency captions according to an embodiment of the present invention. [Figure 4] A schematic diagram showing an audio-visual transformer constructed as a low-latency caption generator combined with a timing detector for low-latency video captioning according to an embodiment of the present invention. [Figure 5] A diagram showing evaluation results obtained by performing a low-latency video captioning test using an audio-visual transformer constructed as a low-latency caption generator combined with a timing detector according to an embodiment of the present invention. [Figure 6] A diagram showing a model architecture according to an embodiment of the present invention, which is composed of a question encoder, an AV encoder, a timing detector, and a response decoder. [Figure 7] A diagram showing a video stream according to an embodiment of the present invention, where the video starts at time TS associated with the ground truth answer Ye' and ends at Te. [Figure 8] A diagram showing process steps executed by a low-latency processing system using a processor and a memory or multiple memory storages storing instructions according to an embodiment of the present invention. [Figure 9] A diagram showing the relationship between the latency ratio and the answer accuracy for MSRVTT-QA according to an embodiment of the present invention. [Figure 10] A diagram showing a comparison of the quality of answer sentences for the AVSD task according to an embodiment of the present invention. [Figure 11A] A diagram showing the distribution of QAs over latency for the AVSD-DSTC7 task with a detection threshold F = 0.3 according to an embodiment of the present invention. [Figure 11B]This figure shows the distribution of QA over latency at a detection threshold F=0.4 for the AVSD-DSTC7 task according to an embodiment of the present invention. [Modes for carrying out the invention]

[0019] While the drawings above illustrate embodiments of the present disclosure, other embodiments are also conceived, as described below. This disclosure presents specific embodiments for illustrative purposes only, not limitation. Those skilled in the art will be able to devise numerous other modifications and embodiments that fall within the scope and spirit of the principles of the embodiments of this disclosure.

[0020] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of exemplary embodiments will provide a practicable description for realizing one or more exemplary embodiments. Various modifications may be made with respect to the function and configuration of the elements without departing from the spirit and scope of the subject matter disclosed as described in the appended claims.

[0021] The following description provides specific details to offer a complete understanding of the embodiments. However, those skilled in the art will understand that embodiments may be carried out without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other cases, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments. Furthermore, similar reference numbers and names in various drawings refer to similar elements.

[0022] Furthermore, individual embodiments may be described as processes shown as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts can describe operations as sequential processes, many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are complete, but it may have additional steps not shown or included in the diagram. Moreover, not all operations in any particular process described will occur in all embodiments. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, the termination of the function may correspond to the function's return to the calling function or main function.

[0023] Furthermore, embodiments of the disclosed subject matter can be implemented at least partially manually or automatically. Manual or automatic implementations can be performed, or at least assisted, by using a machine, hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof. If implemented in software, firmware, middleware, or microcode, program code or code segments for performing the required tasks can be stored in a machine-readable medium. A processor can perform the required tasks.

[0024] The modules and networks illustrated in this disclosure may be computer programs, software, or instruction code that can execute instructions using one or more processors. The modules and networks may be stored in one or more storage devices, or in computer-readable media such as storage media, computer storage media, etc., or in (removable and / or non-removable) data storage devices such as magnetic disks, optical disks, or tapes, etc., the computer-readable media being accessible from one or more processors for executing instructions.

[0025] Computer storage media may include volatile and non-volatile media, removable and non-removable media, implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media may be RAM, ROM, EEPROM, or flash memory, CD-ROM, digital versatile disk (DVD), or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices, or other magnetic storage devices, or any other media that can be used to store desired information and is accessible by applications, modules, or both using one or more processors. Such computer storage media may be part of a device, or accessible or connectable to a device. Any applications or modules described herein may be implemented using computer-readable / computer-executable instructions that can be stored or held by such computer-readable media. (Low-latency captioning)

[0026] According to some embodiments of the present disclosure, a scene captioning system may provide low-latency scene captioning for understanding a scene and describing events in natural language, from real-time monitoring of the scene or online / offline scene streaming.

[0027] In some cases, a scene captioning system can be configured as a video captioning system, which may be referred to as a low-latency video captioning method / system.

[0028] The following provides an exemplary description of a low-latency video scene captioning method / system.

[0029] For example, a stream of signals obtained by a unimodal or multimodal sensor can be converted into captions using natural language. Unimodal is encompassed by multimodal when a single sensor is applied. Multimodal sensors capture information such as visual images, audio signals, 3D localization, thermography, and Wi-Fi. In some cases, the sensor may be an image sensor, microphone, audio-visual signal sensor, LiDAR, mmWave, thermal sensor, olfactory sensor, tactile sensor, or any combination thereof.

[0030] When a low-latency scene captioning method / system is applied to a real-time or offline video stream, the multimodal sensor signal may be a stream of audio-video signals, the scene encoder may be an audio-visual encoder, and the unimodal / multimodal sensor features may be audio features and visual features. Furthermore, the multimodal sensor signal may be a signal / information acquired by the multimodal sensor. For example, the signal may be any of the detected information such as video frames, audio data, 3D localization data, thermal sensor data, olfactory sensor data, tactile sensor data, or any combination thereof.

[0031] Figure 1 is a schematic diagram illustrating a low-latency video captioning process according to an embodiment of the present invention. Assume that a video stream 101 is given as X, an event occurs between time s and time T, and a ground truth caption 102, Y, may be "One of the workers was hit by a bulldozer" regarding the event. A caption is typically a sentence in natural language that describes the event. A prior art video captioning system takes frame x of video stream X. s:T This generates such captions for video clips using the whole, x s:T is video frame x s , x s+1, ..., x T-1 , x T This shows a sequence, each corresponding to a video frame at a given time index. Therefore, such conventional video captioning systems are assumed to function only offline.

[0032] However, such conventional methods are impractical in real-time monitoring or supervision systems where it is essential not only to accurately describe events but also to generate captions as quickly as possible. For this purpose, the low-latency video captioning module 110 captures a portion of the video frame 104, i.e., video frame x from start time s to current time t. s:t The system must have a timing detector 103 that uses a metric to determine whether the current time t is appropriate timing to output a caption. Only if the timing detector 103 detects appropriate timing, the low-latency caption generator 105 generates a caption 106 based on the partial video frame 104. The start time s may be determined based on the timing at which the system generated a previous caption, or the timing at which a specific change in pixel intensity in the video is detected (not shown in Figure 1).

[0033] One remaining problem is the lack of sufficient dataset to train the timing detector 103 and the low-latency caption generator 105, in which case the dataset must be annotated with appropriate early timings and corresponding captions for various videos. The dataset only exists in the case of offline video captioning, i.e., the dataset is annotated at the point when events had already ended in the video. Generally, modifying large amounts of new datasets with such early timing annotations is significantly costly.

[0034] To solve this problem, embodiments of the present invention train a timing detector 103 and a low-latency caption generator 105 using only a dataset for offline video captioning, in which case the detector and generator are optimized not only to accurately describe events but also to generate captions as quickly as possible. Thus, low-latency video captioning is achieved by detecting the appropriate timing to generate the correct caption using only partial video frames 104 up to the current time t. The system can generate a caption 106 that is equivalent to ground truth 102, "One of the workers was hit by a bulldozer," even though it uses only partial video frames 104. On the other hand, conventional methods of training a generator for offline video captioning may generate an incorrect caption 107 for partial video frame 104, such as "A worker is walking," because the generator is trained to generate captions about events that were happening in each video clip. In the low-latency video captioning example 100, the event "One of the workers was hit by a bulldozer" has not yet occurred at time t. Therefore, it is difficult for a prior art system to generate the correct caption, but a system according to the present invention may be able to generate the correct caption because the generator is trained to utilize several signs of a future event within a partial video frame to generate the correct caption. (Low-latency captioning-based dialogue system)

[0035] Figure 2 is a block diagram showing a low-latency video captioning system 200 according to an embodiment of the present invention, which includes a low-latency caption generation training module 300 comprising a training dataset 301, a feature extractor 302, a future frame excluder 303, a pre-trained caption generator 320, a caption generation loss calculator 309, a caption similarity checker 310, and a timing detection loss calculator 311; a low-latency captioning module 110 comprising a timing detector module 103 and a low-latency caption generator 105; and an interaction module 201 comprising a caption understanding module 202 and a response generator 203.

[0036] The low-latency video captioning system 200 includes a human-machine interface (HMI) 210 connectable to a keyboard 211 and a pointing device / medium 212, one or more processors 220, memory 240, a network interface controller (NIC) 250 connectable to a network 290 including a local area network and an internet network, a display and / or speaker interface 260 connectable to a display and / or speaker device 261, a machine interface 262 connectable to a machine actuator 263, multimodal sensor interfaces 271 such as audio interfaces 273 and visual interfaces 275 connectable to input devices including multimodal sensors 272 such as a microphone device 274 and a camera device 276, and a printer interface 280 connectable to a printing device 285. The memory 240 may consist of one or more memory units. The low-latency video captioning-based interaction system 200 can receive multimodal detection data 295 via a network 290 connected to a NIC 250. The storage device 230 includes a feature extractor 302, a low-latency caption generation training module 300, a low-latency captioning module 110, and an interaction module 201. In some cases, the feature extractor 302 may be configured as a multimodal sensor feature extractor 302 when the system is configured as a scene captioning system. Furthermore, the feature extractor 302 may be configured as an audio-visual feature extractor 302 when the system is configured as a video captioning system.

[0037] To perform low-latency captioning, instructions can be sent to a low-latency video captioning system 200 using a keyboard 211, a pointing device / medium 212, or via a network 290 connected to another computer (not shown). The system 200 receives the instructions via the HMI 210 and executes the instructions to perform low-latency video captioning using a processor 220 associated with memory 240 by loading the low-latency captioning module 110.

[0038] The low-latency video captioning module 110 outputs a token sequence as a captioning result for a given multimodal detection feature sequence obtained by the feature extractor 302, and transmits the token sequence to a display / speaker device 265 via the display / speaker interface 260, to a printer device 285 via the printer interface 280, or to another computer (not shown) via the network 290. Each token in the token sequence may be a single word, a single character, a single letter, or a word fragment in text form. (Training procedures for low-latency video captioning systems)

[0039] Figure 3 is a block diagram showing a training module 300 included in a low-latency video captioning system 200, which includes a low-latency captioning module 110 that utilizes a pre-trained caption generator 320 to jointly optimize a low-latency caption generator 103 and a timing detector 105, according to an embodiment of the present invention.

[0040]

number

[0041]

number

[0042] The timing detector 103 may include an encoder 307 and a timing detection module 308. The encoder 307 encodes the future-excluded feature vector sequence Z′ into a hidden activation vector sequence H′. The timing detector 103 may use the encoder 304 of the low-latency caption generator 105 instead of the encoder 307. In this case, the timing detector 103 may not have the encoder 307 and may receive the output of the encoder 304, i.e., H′ may be obtained as H′=H.

[0043]

number

[0044]

number

[0045] If the encoder 304 and decoder 305 are designed as differentiable functions such as a neural network, the parameters of the neural network can be trained using a backpropagation algorithm to minimize cross-entropy loss. Minimizing cross-entropy loss means causing the low-latency caption generator 105 to generate appropriate captions that are as close as possible to ground-truth captions. Furthermore, if the system is configured as a scene captioning system, the encoder 304 may be configured as a multimodal sensor feature encoder 304. If the system is configured as a video captioning system, the encoder 304 may be configured as an audio-visual feature encoder 304.

[0046]

number

number

[0047]

number

[0048] Similar to the low-latency caption generator 105, the timing detector 103 can be trained using a backpropagation algorithm when the encoder 307 and timing detection module 308 are designed as differentiable functions such as neural networks.

[0049]

number

[0050]

number

[0051]

number

[0052]

number

[0053]

number

[0054] This mechanism prompts the timing detector 103 to detect timings in which the low-latency caption generator 105 can generate a caption that is not only close to the ground truth caption Y but also close to the caption Y' generated by the pre-trained caption generator 203 for the entire feature vector sequence Z. This avoids the supervision of the timing detector 103 relying solely on similarity to the ground truth caption Y, thereby promisingly improving the robustness of the timing detector 103. (Several embodiments based on audio-visual transformers)

[0055] Figure 4 is a schematic diagram showing an audio-visual transformer 400 constructed as a low-latency caption generator combined with a timing detector for low-latency video captioning, according to an embodiment of the present invention. The audio-visual transformer 400 for low-latency video captioning includes a feature extractor 401, an encoder 402, a decoder 403, and a timing detection module 404.

[0056] In the case of video stream 405, the feature extractor 401 extracts VGGish features 406 and I3D features 407 from the audio signal 408 and the visual signal 409 from the audio track and the visual track of the video stream 405, respectively. Here, the audio signal 408 can be an audio waveform, and the visual signal 409 can be a sequence of images. The frame rate for feature extraction can be different for each track. The encoder 402 encodes the feature vectors of VGGish 406 and I3D 407. In this case, the sequence of audio features and visual features from the starting point to the current time is supplied to the encoder 402 and is converted into a sequence of hidden activation vectors through the self-attention layer 410, the bimodal attention layer 411, and the feed-forward layer 412. Typically, this encoder block 413 is repeated N times (e.g., N = 6 or more). The final sequence of hidden activation vectors is obtained through the Nth encoder block.

[0057]

Number

[0058]

Number

[0059]

Number

[0060] The self-attention layer 410 extracts the temporal dependencies within each modality, where all the arguments for MHA() are the same, i.e., A n-1 or V n-1The bimodal attention layer 411 further extracts cross-modal dependencies between audio and visual features, in this case taking keys and values ​​from the other modality. Then, the feedforward layer 412 is applied point by point. The encoded representations for the audio and visual features are A N and V N It is obtained as follows.

[0061] The timing detection module 404 receives an encoded hidden vector sequence based on the audio-visual information available at that time. The role of the timing detection module 404 is to estimate whether the system should generate a caption for a given encoded feature. The timing detection module 404 first processes the encoded vector sequences from each modality in which the 1D convolutional layers 414 are stacked as follows:

number

[0062] Next, each time convolution sequence is aggregated into a single vector through the operations of pooling 415 and concatenation 416.

number

[0063]

number

[0064] The timing detection module 404 detects a probability that is higher than the threshold, for example, P(d=1|X A ,X V If )>0.5, the decoder provides the encoded hidden vector sequence (A N ,V N Generate captions based on ).

[0065] Decoder 403 is the starting word ( <sos>The decoder iteratively predicts the next word from the given word. In each iteration step, the decoder receives a pre-generated word 419 and estimates the posterior probability distribution of the next word 420 by applying a word embedding 421, M decoder blocks 422, a linear layer 423, and a softmax operation 424.

[0066]

number

[0067]

number

[0068]

number

[0069] Multiple words with the highest probability can be selected, and multiple caption candidates can also be considered according to the beam search technique in the search module 306. (training)

[0070] The multimodal encoder, timing detector, and caption decoder are trained together, so that the model achieves caption quality comparable to the full video, even when a given video is shorter than the original video due to truncation of the latter half.

number

[0071] Figure 5 shows the evaluation results obtained by performing a video captioning benchmark according to an embodiment of the present invention.

[0072] The proposed low-latency caption generation was tested using the ActivityNet Captions dataset (Krishna et al. (2017)), which consists of 100,000 caption sentences associated with transient localization information based on 20,000 YouTube® videos. The dataset was split into 50%, 25%, and 25% for training, validation, and testing. The validation set was split into two subsets, and the performance for these two subsets is reported. The mean duration of the video clips was 35.5 seconds, 37.7 seconds, and 40.2 seconds for the training set and validation subsets 1 and 2, respectively. VGGish and I3D features were used. The VGGish feature was configured to form a 128-dimensional vector sequence for the audio track of each video, with each audio frame corresponding to a unique 0.96-second segment. The I3D feature was configured to form a 1024-dimensional vector sequence for the video track, with each visual frame corresponding to a unique 2.56-second segment.

[0073] First, a multimodal transformer was trained on the entire video clip and its ground-truth captions. This model was used as the baseline and training model. N=2 was used for the encoder block, M=2 for the decoder block, and there were 4 attention heads. The vocabulary size was 10172, and the dimension of the word embedding vectors was 300.

[0074] The proposed model for online captioning was trained using incomplete video clips according to the steps of the present invention. The training process used α=β=γ=1 / 3 for the loss function. The dimensions of the hidden activations in the audio attention layer and visual attention layer were 128 and 1024, respectively. The timing detector had two stacked 1D convolutional layers with ReLU nonlinearity between them. Performance was measured by BLEU3, BLEU4, and METEOR scores.

[0075] The latency ratio represents the ratio of the video duration used for captioning to the duration of the original video clip. In the baseline model, the latency ratio is always 1, which means that all frames are used to generate the caption.

[0076] Figure 5 compares captioning methods in terms of METEOR score for validation subset 1. The model selected for evaluation was trained with (empirically determined) S=0.6 and had the best METEOR score for validation subset 2. Latency was controlled with a detection threshold F. As shown in Figure 5, the proposed method at 55% latency achieves a METEOR score of 10.45, which is 98% of the baseline score of 10.67, with only slight degradation. It also achieves a METEOR score of 10.00 at 28% latency, which is 94% of the baseline. We tested a naive method that takes video frames from the beginning at a fixed ratio relative to the original video length and performs baseline captioning on the shortened video clip. The results show that, at comparable latency, the proposed approach is clearly superior to the naive method.

[0077] The table also includes results using a unimodal transformer that receives only visual features. These results show that the proposed method works only for visual features, but its performance is degraded due to the lack of audio features. This result indicates that audio features are essential even for the proposed low-latency method.

[0078] Low-latency captions are input into a dialogue system trained with caption-action command pairs to understand the scene and determine the next action. The dialogue system may use an inter-sequence neural network model to translate the received captions into sequences of words for human responses or sequences of action commands for controlling a robot.

[0079] Furthermore, several embodiments are based on the recognition that scene-aware interaction technologies enable machines to interact with humans based on shared knowledge gained by recognizing and understanding their surroundings using various types of sensors. Audio-visual scene-aware dialog (AVSD) is one such scene-aware interaction technology. According to some AVSD-based recognitions, end-to-end methods can better handle flexible conversations between users and systems by training models on large conversation datasets. Such methods have been extended to engage in conversations about objects and events happening around the machine or user, based on an understanding of the dynamic scene captured by multimodal sensors such as video cameras and microphones. This extension allows users to ask questions about what is happening around them. For example, this framework can be applied to visual question answering (VQA) for one-shot QA about still images, visual dialogue where AI agents engage in meaningful conversations with humans about still images using natural conversational language, video QA for one-shot QA about video clips, and AVSD for QA-based conversations with humans about video clips. Video QA is a QA (question answering) task based on answering single questions about video clips, such as those found on YouTube, and is standardized with the MSVD (Microsoft Research Video Description Corpus) and MSRVTT (Microsoft Research Video to Text) datasets. AVSD is a multi-turn dialogue task based on generating responses to user questions about video clips from everyday life, and is represented by the Dialog Systems Technology Challenges (DSTC).The AVSD system is trained to generate sentences that answer user questions about video clips. In this case, the system needs to understand what events occurred, when, how, and by whom, based on time-series audiovisual features, in order to provide the correct answers.

[0080] At any given moment, countless events occurring in the real world are captured by cameras and stored as massive video data resources. Since the S2VT (Sequence to Sequence -- Video to Text) system was first proposed, video captioning has been actively researched in the field of computer vision, using an end-to-end sequence-to-sequence model, to effectively retrieve such recordings regardless of whether they are offline or online. The goal is to generate video descriptions (captions) about objects and events within video clips. To further leverage audio features to identify events, multimodal attention schemes enable the fusion of audio and visual features such as VGGish (a speech classification model similar to Visual Geometry Group (VGG)) and I3D (Interactive Three Dimensional) to generate video captions. Such video clip captioning techniques have been extended to offline video stream captioning techniques such as high-density video captioning and progressive video description generators, in which case all notable events in the video stream are temporally localized, and event-triggered captions are generated in a multithreaded manner. While all video captioning technologies have been based on LSTM until now, transformers can be successfully applied in combination with the audio-visual attention framework.

[0081] Current dialogue systems for video QA typically process offline video data and generate answers to questions after the clip ends. For dialogue systems to enable real-time conversations about scenes for surveillance and robotics applications, they need to understand scenes and events and respond quickly to user queries about those scenes. In this study, an audio-visual transformer was tested using the ActivityNet Captions dataset within an offline video captioning system and achieved the best performance for high-density video captioning tasks.

[0082] However, such scene-aware interaction tasks assume that the system has access to every frame of the video clip when predicting an answer to a question. This assumption is impractical for real-time dialogue systems that monitor an ongoing audiovisual scene, for which it is essential not only to accurately predict answers but also to respond to the user as quickly as possible by quickly discovering question-related events within the online video stream and generating appropriate answers. Such functionality requires the development of new low-latency QA techniques.

[0083] In previous research, we proposed a low-latency audio-visual captioning method that can accurately and quickly describe events without waiting for the end of a video clip, optimizing captioning timing. In parallel, we recently introduced a novel AVSD task for the 3rd AVSD Challenge at DSTC10, which specifically requires the system to demonstrate temporal inference by discovering evidence from video to support its answer. This is based on a new extension of the DSTC10 AVSD dataset, for which we collected human-generated temporal inference data. This study proposes extending our low-latency captioning method to a scene-aware interaction task and combining it with our inference AVSD system to develop a low-latency online scene-aware interaction system.

[0084] Some extensions to research on low-latency video captioning can construct novel methods that can optimize the timing of each response generation while balancing response generation latency and response quality. In the case of video QA, the timing detector now plays the role of discovering the timing of events relevant to the question, rather than determining when sufficient understanding was gained to generate a generic caption, as in the case of video captioning. The audio-visual scene-aware dialogue system built for the 10th Dialog System Technology Challenge can be extended to take advantage of low-latency capabilities. As an example, experiments with the MSRVTT-QA and AVSD datasets show that our approach achieves 97%–99% of the upper limit of response quality given by a transformer pre-trained using the entire video clip, using less than 40% of the initial frames.

[0085] In the same vein as the low-latency captioning technique, another system according to a different embodiment optimizes output timing for each response based on a trade-off between latency and response quality. We train a low-latency audio-visual transformer consisting of (1) a transformer-based response generator that attempts to mimic the output of a similar pre-trained response generator capable of viewing the entire video, while attempting to generate a ground-truth response after viewing only a small portion of all video frames, and (2) a CNN-based timing detector that can discover the best timing to output a response to an input question so that the responses ultimately generated by the two transformers are sufficiently similar to each other. The proposed, jointly trained response generator and timing detector can generate a response as soon as relevant events occur, even at the early stages of a video clip, and can even predict future frames. Thanks to the combination of information from multiple modalities, the system has more opportunities to recognize events at an earlier timing by relying on the earliest cue in one of the modalities. Experiments using MSR-VTT QA and AVSD datasets demonstrate that our approach enables low-latency video QA with response quality comparable to offline video QA baselines that utilize the entire video frame.

[0086] According to one embodiment of the present invention, video QA is provided that differs from video captioning, generating appropriate answers to user questions as quickly as possible. To this end, we introduce a question encoder to provide question embeddings to a timing detector and extend the text generator to accept questions as contextual information. Thus, the proposed method utilizes the same underlying mechanism for low-latency processing, but the model is extended for video QA tasks. (Low-latency video QA model)

[0087] We build our proposed model for low-latency video QA on a DSTC10-AVSD system employing an AV transformer architecture. For the DSTC10-AVSD challenge, we extended the AV transformer with student-teacher collaborative learning and attention multimodal fusion to achieve state-of-the-art performance. Like our low-latency video captioning system, the low-latency QA model receives video and audio features in a streaming manner, and a timing detector determines when to generate a response for the feature sequence the model has received up to that moment. Figure 6 shows a model architecture (low-latency video QA architecture) 600 for low-latency video QA according to some embodiments of the present invention. The model architecture 600 includes functional parts similar to those used in the audio-visual transformer 400. In the drawings, the same numbers are used for parts (or layers) corresponding to parts (or layers) used in the audio-visual transformer 400. The low-latency video QA architecture 600 further includes word embedding 621, a question encoder (text encoder) 613 having a self-attention layer 610 and a feedforward layer 612, a pooling 615 connected to contact 416, an AV encoder 402, a timing detector 404, and a response decoder 403, the AV encoder 402 being shared by the timing detector 404 and the response decoder 403.

[0088] Furthermore, the formulas or equations used to describe the Audio-Visual Transformer 400 are used in the following description to describe the Low-Latency Video QA Architecture 600.

[0089] Given a video stream and question text as input, the AV encoder encodes VGGish and I3D features extracted from the audio and video tracks, while a transformer-based text encoder encodes the question. A sequence of audio and visual features from the starting point to the present is given to the encoder and transformed into a hidden vector sequence through a self-attention layer, a bimodal attention layer, and a feedforward layer. This encoder block is repeated N times, and the final encoded representation is obtained through the Nth encoder block. The question word sequence is also encoded through a word embedding layer, followed by a transformer with N' blocks.

[0090]

number

[0091]

number

[0092]

number

[0093]

number

[0094] If the timing detector outputs a probability higher than the threshold, for example, if P(d=1|A,V,Q)>0.5, the decoder generates a response based on the encoded representation.

[0095]

number

[0096] The AV encoder, question encoder, response decoder, and timing detector are trained together so that the system can achieve response quality comparable to a complete video, even when a given video is shorter than the original due to the latter half being truncated.

[0097]

number

[0098]

number

[0099]

number

[0100]

number

[0101]

number

[0102]

number

[0103] Note T S Let's assume that this has already been decided.

[0104] According to several embodiments of the present invention, an artificial intelligence (AI) low-latency processing system is provided. Figure 8 shows a computer-implemented method 800, which includes process steps performed by the low-latency processing system using a processor and memory or memory storage, according to one embodiment of the present invention.

[0105] A low-latency processing system may include a processor and memory storing instructions as method 800 implemented by a computer. When an instruction is executed by the processor, it causes the low-latency processing system to collect a sequence of frames. The sequence of frames jointly contains information distributed across at least some of the frames in the sequence. Furthermore, the instruction causes a timing neural network to be executed, which is trained to identify a subsequence of early frames in the sequence of frames that contains at least a portion of the information that indicates the information, and a decoding neural network to be executed, which is trained to decode information from the portion of information in the subsequence of frames. The timing neural network is trained jointly with the decoding neural network to repeatedly identify a minimum number of subframes from the beginning of the training sequence of frames that contain a portion of the training information sufficient to decode the training information.

[0106] In some cases, the timing detector neural network is trained in conjunction with the decoder neural network on features of different subsequences of a frame sequence to minimize a multitask loss function that includes a time detection loss and an information generation loss. The multitask loss function may include three losses, which define (1) the accuracy of the decoded information, (2) the difference between the information decoded from a subsequence of frames and the information decoded from the entire frame sequence, and (3) the accuracy of the timing detector neural network's prediction. Furthermore, the processor may be configured to extract features from each frame of the frame sequence by running a feature extractor neural network, encode the extracted features from each frame by running a feature encoder neural network to generate a sequence of encoded features, provide the sequence of encoded features to the timing detector neural network to identify the subsequence of encoded features that represent the subsequence of frames, and provide the subsequence of encoded features to the decoder neural network to decode the information. In other cases, the processor may trigger the execution of a module of the AI ​​low-latency processing system when it receives a new input frame attached to a sequence of frames. The information may be an audio scene, a video scene, or captions for an audio-video scene. The information may also be answers to questions about the sequence of frames. In some cases, the frames may contain multi-model information from different sensors of different modalities. Furthermore, the information may also be answers to questions about the sequence of frames, and the processor may run a text encoder neural network to encode the questions, feed the encoded questions to a timing neural network, and feed the questions or encoded questions to a decoding neural network. (experiment)

[0107] We evaluate our low-latency video QA method using the MSRVTT-QA and AVSD datasets. MSRVTT-QA is based on the MSR-VTT dataset, which contains 10,000 video clips and 243,000 question-answer (QA) pairs. QA pairs are automatically generated from manually annotated captions for each video clip, where the question is a sentence and the answer is a single word. We divide the dataset according to the MSR-VTT dataset, with 65% training, 5% validation, and 30% testing. AVSD is a set of text-based dialogues about short videos from the Charades dataset, each consisting of untrimmed multi-action videos with an audio track. In AVSD, two parties, a named questioner and a respondent, engage in dialogue about events in the provided video. The respondent, having already watched the video, is tasked with answering questions from the questioner. We followed the AVSD challenge setup, in which the training, validation, and test sets consisted of 7.7k, 1.8k, and 1.8k dialogues, respectively, with each dialogue containing 10 rounds of Q&A pairs, where both the questions and answers were in text form. The video clip lengths ranged from 10 to 40 seconds.

[0108] The VGGish feature was configured to form a 128-dimensional vector sequence for each video's audio track, in which case each audio frame corresponds to a non-overlapping 0.96-second segment. The I3D feature was configured to form a 2048-dimensional vector sequence for each video track, in which case each visual frame corresponds to a non-overlapping 2.56-second segment.

[0109] We first trained a multimodal transformer with all video clips and QA pairs. This model was used as the baseline and training model. We used N=2 audio-visual encoder blocks, N'=4 question encoder blocks, and M=4 decoder blocks, setting the number of attention heads to 4. The vocabulary size was 7,599 for MSRVTT-QA and 3,669 for AVSD. The dimension of the word embedding vectors was 300.

[0110] The proposed model for low-latency video QA was trained on incomplete video clips following the steps in Section 3.2. The architecture was the same as the baseline / supervised model, except for the addition of a timing detector. Throughout the training process, the loss function α=β=γ=1 / 3 was consistently used, and the threshold S=0.9 from equation (24) was used. The dimensions of the hidden activations in the audio and visual attention layers were set to 256 and 1024, respectively, the dropout rate was set to 0.1, and label smoothing techniques were applied. The timing detector consisted of two stacked 1D convolutional layers with ReLU nonlinearity in between. Performance was measured by MSRVTT-QA answer accuracy, as well as AVSD BLEU4 and METEOR scores.

[0111] Figure 9 shows the relationship between latency ratio and response accuracy for MSRVTT-QA. Latency ratio represents the ratio of frames actually used (from the beginning) to the total number of video frames. Baseline results were obtained by simply omitting future frames at various ratios using the baseline (trained) model. Results for the proposed model were obtained by changing the detection threshold F. MSRVTT-QA accuracy represents the percentage of one-word responses that match ground truth. These results demonstrate that our proposed method achieves low-latency video QA with a much smaller degradation of accuracy compared to the baseline. Our approach achieves 97% of the upper limit of response quality given by a transformer pre-trained using the entire video clip, using only the first 40% of the frames.

[0112] Figure 10 shows a comparison of response quality for the AVSD task. Latency ratios were controlled by setting a detection threshold F so that the proposed method yielded average ratios of 1.0, 0.5, and 0.2. Future frames with the above fixed ratios were excluded for the baseline system. As shown in the table, our proposed method is slightly better than the baseline even at a latency of 1.0. This may be due to increased model robustness achieved by training with randomly shortened videos. In addition, the proposed method maintains the same level of BLEU4 and METEOR scores at a latency of 0.5, achieves competitive scores even at a latency of 0.2 with slight degradation, and reaches 98%-99% of the score under the latency of 1.0.

[0113] Figures 11A and 11B show the distribution of QAs across latencies at detection thresholds F=0.3 and F=0.4 for the AVSD-DSTC7 task, corresponding to the results for latencies 0.2 (left) and 0.5 (right) in Figure 10. These results indicate that most AVSD QAs require either only the initial frame or all frames to generate an accurate answer. We investigated the reasons for the polarized distribution. The most frequent question pattern leading to early decisions is "How does the video begin?". Additionally, the training data contains several fixed answers, such as "One" to "How many people are in the video?". Such common linguistic patterns can also cause early decisions. Cases where decisions are delayed include patterns such as "How does the video end?". Such questions are natural for questioners who need to generate video captions through 10 QAs without viewing the entire video.

[0114] According to some of the embodiments described above, the low-latency video QA method can accurately and quickly answer user questions without waiting for the end of the video clip. The proposed method optimizes the timing of outputting each answer based on the trade-off between latency and answer quality. The system can generate answers in the early stages of a video clip using the MSRVTT-QA and AVSD datasets, achieving 97% to 99% of the upper limit of answer quality given by a pre-trained transformer using the entire video clip, using less than 40% of the initial frames.< / sos>

Claims

1. An artificial intelligence (AI) low-latency processing system, the AI ​​low-latency processing system comprising a processor and a memory storing instructions, wherein when an instruction is executed by the processor, the AI ​​low-latency processing system The system performs the task of collecting a sequence of frames, the sequence of frames collectively including information distributed across at least some of the frames in the sequence of frames, and further, Running a timing detector neural network trained to identify an early subsequence of frames in a sequence of frames that includes at least a portion of the information relating to the aforementioned information, This involves running a decoder neural network that has been trained to decode the information from a portion of the information in a subsequence of the frame, The aforementioned information is an answer to a question regarding the captions of an audio scene, a caption of a video scene, a caption of both an audio and video scene, or a sequence of frames. The frame includes multi-model information coming from different sensors of different modalities. The timing detector neural network is trained in conjunction with the decoder neural network to repeatedly identify a minimum number of subframes from the beginning of the training sequence of frames, each containing a portion of the training information sufficient to decode the training information. The timing detector neural network is trained in conjunction with the decoder neural network on features of different subsequences of the frame sequence to minimize a multitask loss function that includes time detection loss and information generation loss. The multitask loss function includes three losses, the three losses of which determine (1) the accuracy of the decoded information, (2) the difference between the information decoded from a subsequence of the frame and the information decoded from the entire sequence of the frame, and (3) the prediction accuracy of the timing detector neural network, in an AI low-latency processing system.

2. The aforementioned processor, By running a feature extractor neural network, features are extracted from each frame in the sequence of frames. By executing a feature encoder neural network, the extracted features of each frame are encoded to generate a sequence of encoded features. The sequence of encoded features is provided to the timing detector neural network to identify the subsequence of encoded features that represent the subsequence of the frame. The AI ​​low-latency processing system according to claim 1, configured to decode the information by providing a subsequence of the encoded features to the decoder neural network.

3. The AI ​​low-latency processing system according to claim 1, wherein the processor, upon receiving a new input frame attached to the sequence of frames, triggers the execution of a module of the AI ​​low-latency processing system.

4. The information is the answer to the question regarding the sequence of frames, The aforementioned processor, The text encoder neural network is executed to encode the aforementioned question, The encoded question is given to the timing detector neural network, The AI ​​low-latency processing system according to claim 2, configured to provide the aforementioned question or the encoded question to the decoder neural network.

5. A computer-implemented method for an artificial intelligence (AI) low-latency processing system, the AI ​​low-latency processing system comprising a processor and a memory storing instructions for the computer-implemented method, the steps of which are performed using the processor, This includes collecting a sequence of frames, wherein the sequence of frames jointly includes information distributed across at least some of the frames in the sequence of frames, and further, Running a timing detector neural network trained to identify an early subsequence of frames in a sequence of frames that includes at least a portion of the information relating to the aforementioned information, This includes running a decoder neural network trained to decode the information from a portion of the information in a subsequence of the frame, The aforementioned information is an answer to a question regarding the captions of an audio scene, a caption of a video scene, a caption of both an audio and video scene, or a sequence of frames. The frame includes multi-model information coming from different sensors of different modalities. The timing detector neural network is trained in conjunction with the decoder neural network to repeatedly identify a minimum number of subframes from the beginning of the training sequence of frames, each containing a portion of the training information sufficient to decode the training information. The timing detector neural network is trained in conjunction with the decoder neural network on features of different subsequences of the frame sequence to minimize a multitask loss function that includes time detection loss and information generation loss. The multitask loss function includes three losses, the computer-implemented method for determining (1) the accuracy of the decoded information, (2) the difference between the information decoded from a subsequence of the frame and the information decoded from the entire sequence of the frame, and (3) the prediction accuracy of the timing detector neural network.

6. The aforementioned processor, By running a feature extractor neural network, features are extracted from each frame in the sequence of frames. By executing a feature encoder neural network, the extracted features of each frame are encoded to generate a sequence of encoded features. The sequence of encoded features is provided to the timing detector neural network to identify the subsequence of encoded features that represent the subsequence of the frame. A computer-based method according to claim 5, configured to provide a subsequence of the encoded features to the decoder neural network to decode the information.

7. The computer-implemented method according to claim 5, wherein when the processor receives a new input frame attached to the sequence of frames, it triggers the execution of a module of the AI ​​low-latency processing system.

8. The information is the answer to the question regarding the sequence of frames, The aforementioned processor, The text encoder neural network is executed to encode the aforementioned question, The encoded question is given to the timing detector neural network, A method implemented by a computer according to claim 6, configured to provide the aforementioned question or the encoded question to the decoder neural network.

Images