Method and device for following sound objects

Abstract

The present disclosure provides a method and device for sound object following. The present disclosure relates to a method and device for processing multimedia signals. More specifically, the present disclosure relates to a method and device thereof, the method comprising: obtaining at least one video object from a multimedia signal and obtaining at least one audio object from a multimedia signal; extracting video feature information for at least one video object and audio feature information for at least one audio object; and determining the correlation between at least one video object and at least one audio object through an object matching engine based on the video feature information and the audio feature information.

Description

Technical Field

[0001] The present disclosure relates to multimedia signal processing, and more particularly, to a method and apparatus for providing a sound having a three-dimensional sound effect and an immersive feeling through an apparatus configured to output a video signal and an audio signal. Background Art

[0002] With the advancement of technology, devices equipped with larger and higher resolution displays and multiple speakers have become widespread. In addition, research on video coding technology for sending and receiving more vivid images and audio coding technology for sending and receiving immersive audio signals has been actively conducted, and multimedia content generated or created based on video coding technology and audio coding technology is widely circulated. For example, content generated and supported based on high-efficiency video coding (HEVC) or H.265 standards is being distributed. In addition, content ranging from stereo-based content to content supporting multi-channel or object-based audio standards such as Dolby Atmos (or content generated based on object-based audio standards) is being distributed.

[0003] However, in the case of conventional three-dimensional (3D) audio processing technology, the encoder transmits information about the position and motion of the sound source or the video object related to the audio to the decoder through the audio bitstream, and the decoder processes the audio signal to produce a 3D effect based on the information operation about the position and motion contained in the audio bitstream. Therefore, in the mixing process of producing audio content based on the 3D audio processing technology, the audio mixing technician locates the sound source or the video object related to the audio, and generates mixing parameters for providing a sense of space based on the position of the sound source or the video object. Then, the generated parameters are encoded in the bitstream together with the object to generate audio content. Audio professionals need to measure and adjust the position and motion of the sound source or the video object related to the audio very accurately, and generate information about them to maximize the 3D effect during the encoding process. Therefore, it may take a lot of time and money to create content based on conventional 3D audio processing technology. In addition, it may be difficult to apply conventional 3D audio processing technology to broadcast and real-time streaming content.

[0004] Therefore, 3D audio processing technologies such as Dolby Atmos are limited to some content such as movies, and have not yet been applied to most broadcast and streaming content. For audio content contained in most broadcast and streaming content, the sound image cannot be correctly positioned according to the position of the sound source or video object, and therefore, the resulting immersion is limited. Summary of the invention

[0005] Accordingly, the present disclosure is directed to a method and apparatus for sound object following that substantially obviates one or more problems due to limitations and disadvantages of the related art.

[0006] An object of the present disclosure is to provide a method and apparatus for positioning a sound image of an audio object based on a position of a video object related to the audio object to improve immersion with respect to audio.

[0007] Another object of the present disclosure is to provide a method and apparatus for effectively determining a relationship between an audio object and a video object contained in a multimedia signal.

[0008] Those skilled in the art will recognize that the objectives that can be achieved by the present disclosure are not limited to what has been specifically described above, and the above and other objectives that can be achieved by the present disclosure will be more clearly understood from the following detailed description.

[0009] To achieve the purpose of the present disclosure, in a first aspect of the present disclosure, a method for processing a multimedia signal by a device is provided, the method comprising: obtaining at least one video object from the multimedia signal and obtaining at least one audio object from the multimedia signal; extracting video feature information for at least one video object and audio feature information for at least one audio object; and determining, based on the video feature information and the audio feature information, a correlation between at least one video object and at least one audio object by an object matching engine.

[0010] To achieve the purpose of the present disclosure, in a second aspect of the present disclosure, a device is provided herein, which is configured to process a multimedia signal, and the device includes: a memory, which stores instructions; and at least one processor, which is operably coupled to the memory and is configured to implement the following operations when executing the instructions: obtaining at least one video object from the multimedia signal and obtaining at least one audio object from the multimedia signal; extracting video feature information for at least one video object and audio feature information for at least one audio object; and determining the correlation between at least one video object and at least one audio object through an object matching engine based on the video feature information and the audio feature information.

[0011] Additionally or alternatively, determining the correlation between at least one video object and at least one audio object may include: obtaining, by an object matching engine, information about the relationship between each of the at least one video object and a specific audio object of the at least one audio object based on video feature information and audio feature information; and determining, from among the at least one video object, a specific video object related to the specific audio object based on the information about the relationship between each of the at least one video object and the specific audio object.

[0012] Additionally or alternatively, determining the specific video object related to the specific audio object may include determining the video object related to the value of the information about the relationship as the specific video object based on the value of the information about the relationship being greater than a predetermined value.

[0013] Additionally or alternatively, determining the specific video object related to the specific audio object may further include determining a video object related to a maximum value of information about a relationship from among the multiple video objects as the specific video object based on at least one video object including the multiple video objects.

[0014] Additionally or alternatively, determining the specific video object related to the specific audio object may include determining the video object related to the value of the information about the relationship as the specific video object based on the value of the information about the relationship being less than a predetermined value.

[0015] Additionally or alternatively, determining the specific video object related to the specific audio object may further include determining a video object related to a minimum value of information about a relationship from among the multiple video objects as the specific video object based on at least one video object including the multiple video objects.

[0016] Additionally or alternatively, the information about the relationship may have real valued values.

[0017] Additionally or alternatively, video feature information may be extracted based on the vertical length and horizontal length of a lip skeleton.

[0018] Additionally or alternatively, video feature information may be extracted based on: the ratio of the distance between a top boundary of a first rectangular area including a face and a top boundary of a second rectangular area including lips within the first rectangular area, the height of the second rectangular area, and the distance between a bottom boundary of the second rectangular area and a bottom boundary of the first rectangular area; and the ratio of the distance between a left boundary of the second rectangular area and a left boundary of the first rectangular area, the width of the second rectangular area, and the distance between a right boundary of the second rectangular area and a right boundary of the first rectangular area.

[0019] Additionally or alternatively, audio feature information may be extracted based on linear prediction coding (LPC).

[0020] Additionally or alternatively, audio feature information may be extracted based on a log-Mel filters-of-bank.

[0021] Additionally or alternatively, audio feature information may be extracted based on Mel-Frequency Cepstral Coefficients (MFCC).

[0022] Additionally or alternatively, the audio feature information may include onset information about at least one audio object.

[0023] Additionally or alternatively, the object matching engine may include a model trained based on learning.

[0024] As apparent from the following detailed description, the present disclosure can provide the following advantageous technical effects.

[0025] According to the present disclosure, the sense of immersion regarding audio may be improved by positioning a sound image of an audio object based on a position of a video object related to the audio object.

[0026] In addition, according to the present disclosure, the relationship between audio objects and video objects contained in a multimedia signal can be effectively determined.

[0027] Those skilled in the art will recognize that the effects that can be achieved through the present disclosure are not limited to the contents that have been specifically described above, and other advantages of the present disclosure will be more clearly understood from the following detailed description in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this application, illustrate embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. In the drawings:

[0029] Figure 1 An exemplary flow chart illustrating the method proposed in the present disclosure;

[0030] Figure 2 An example of obtaining audio frames and video frames from a multimedia signal is illustrated;

[0031] Figure 3 An example of illustrating obtaining at least one audio object from an audio frame;

[0032] Figure 4 An example of illustrating obtaining at least one video object from a video frame;

[0033] Figure 5 A block diagram illustrating sound object following according to the proposed method of the present disclosure;

[0034] Figure 6 An example of performing audio / video classification is shown;

[0035] Figure 7 and Figure 8 An example of performing feature extraction on a video object is shown;

[0036] Fig. 9 illustrates an example of the operation of an object matching engine;

[0037] Fig.10 illustrates an example of sound image positioning according to the screen position of a video object;

[0038] Fig.11 and Fig.12 An example of the operation of a sound object following is illustrated;

[0039] Fig.13 A block diagram illustrating an apparatus to which the present disclosure is applicable;

[0040] Fig.14 A more detailed block diagram illustrating a device to which the present disclosure is applicable;

[0041] Fig.15 Another example of a wireless device applicable to the present disclosure is shown;

[0042] Fig.16 The handheld device used in the present disclosure is illustrated;

[0043] Fig.17 A vehicle or an autonomous driving vehicle to which the present disclosure is applied is shown;

[0044] Fig.18 The illustration shows a vehicle applicable to the present disclosure;

[0045] Fig.19 The illustration shows an XR device applicable to the present disclosure; and

[0046] Fig. 20 The diagram illustrates an artificial intelligence (AI) device applied to the present disclosure. DETAILED DESCRIPTION

[0047] Reference will now be made in detail to the preferred embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or like parts.

[0048] In order to apply 3D audio processing technology such as Dolby Atmos to broadcast or real-time streaming content, audio mixing technicians need to generate and send mixing parameters for 3D effects in real time. Current technology is difficult to perform real-time processing. In particular, in order to correctly apply 3D audio processing technology such as Dolby Atmos, it is necessary to accurately identify the position of the speaker set on the user (or decoder) side. However, content producers and suppliers (or encoders) rarely have the possibility to identify all information about the position of the speaker in a typical house. Therefore, there are technical difficulties in applying conventional 3D audio processing technology to the audio content included in broadcast or real-time streaming content. In addition, the audio content (or content not produced or created based on 3D audio processing technology) contained in the content that cannot be obtained by applying conventional 3D audio processing technology provides a greatly reduced sense of immersion because its sound image is not correctly positioned according to the position of the sound source or video object.

[0049] This document proposes a method for positioning a sound image based on the position of a sound source or video object related to audio content contained in content obtained by applying 3D audio processing technology (or content not generated or created based on 3D audio processing technology) to improve immersion. Specifically, in this specification, a method for sound object following is proposed. Sound object following may refer to a technique for identifying the position of a thing (or object) or a sound source emitting sound in an image or video and controlling / adjusting the audio output to achieve sound image positioning based on the position of the thing (or object) or the sound source emitting sound.

[0050] The method for sound object following proposed in this specification is a technology for improving three-dimensional sound effects and immersion in a device including a display and a speaker. In the method proposed in this specification, the relationship between a single image object (or video object) or multiple image objects (or video objects) and a single sound object (or audio object) or multiple sound objects (or audio objects) is determined, and then the output rate (or gain) is controlled and / or adjusted so that the sound image of the relevant sound object (or audio object) is positioned at the position of the image object (video object).

[0051] By using the proposed method of the present disclosure, even when playing back content that has not undergone conventional 3D audio processing technology, the sound source or image object (or video object) emitting the sound can be identified, and the relevant audio signal can be output so that the sound image is positioned at the location of the sound source or image object (or video object). Therefore, the sense of immersion can be improved. When the proposed method of the present disclosure is applied, its effect can be further enhanced in devices equipped with large screens or multi-channel speakers.

[0052] Artificial Intelligence (AI)

[0053] The proposed method of the present disclosure can be implemented based on AI. AI refers to the field of studying artificial intelligence or the methodology that can create artificial intelligence, while machine learning refers to the field of defining various problems in the field of AI and studying the methodology for solving the problems. Machine learning is defined as an algorithm that improves task performance through consistent task experience.

[0054] An artificial neural network (ANN) is a model used in machine learning and may refer to an overall problem-solving model that consists of artificial neurons (nodes) that form a network through a combination of synapses. An ANN may be defined by the connection pattern between neurons in different layers, the learning process that updates model parameters, and the activation function that generates output values.

[0055] ANN may include an input layer and an output layer, and optionally one or more hidden layers. Each layer may include one or more neurons, and ANN may include synapses connecting one neuron to another neuron. In ANN, each neuron may output a function value of an activation function of an input signal input through a synapse, weights, and biases.

[0056] Model parameters refer to parameters determined by learning and include the weights of synaptic connections and the biases of neurons. Hyperparameters refer to parameters to be set in a machine learning algorithm before learning and include the learning rate, number of repetitions, minimum batch size, and initialization function.

[0057] The learning objective of ANN can be to determine the model parameters that minimize the loss function. The loss function can be used as an indicator to determine the optimal model parameters during the learning process of ANN.

[0058] Machine learning can be categorized into supervised learning, unsupervised learning, and reinforcement learning based on the learning scheme.

[0059] Supervised learning may refer to a method of training an ANN by labels for the data given for learning. Here, labels may refer to the correct answer (or result value) that the ANN must infer when the learning data is input to the ANN. Unsupervised learning may refer to a method of training an ANN without labels for the data given for learning. Reinforcement learning may refer to a learning method that enables an agent defined in an environment to learn to select an action or series of actions that maximizes the cumulative reward in each state.

[0060] Machine learning implemented by a deep neural network (DNN) including multiple hidden layers among ANNs is called deep learning, which is a part of machine learning. In the following description, machine learning is used as a concept including deep learning.

[0061] Extended Reality (XR)

[0062] The proposed method of the present disclosure is applicable to extended reality (XR). XR is collectively referred to as virtual reality (VR), augmented reality (AR) and mixed reality (MR). VR technology only provides computer graphics (CG) images of real-world objects, backgrounds, etc., and AR technology provides virtual CG images on images of real objects. MR technology is a CG technology that can mix and combine virtual objects in the real world.

[0063] MR technology is similar to AR technology in that it displays both real and virtual objects. However, in AR technology, virtual objects are used in a way that supplements real objects, while in MR technology, virtual objects and real objects have the same properties.

[0064] XR technology is applicable to head-mounted displays (HMD), head-up displays (HUD), mobile phones, tablets, laptops, desktop PCs, TVs, digital signage, etc. Devices that apply XR technology can be called XR devices.

[0065] Proposed method

[0066] Figure 1 is an exemplary flow chart of the method proposed in the present disclosure. Figure 1 The flowchart illustrated in FIG. 1 is a non-limiting example, and the proposed method of the present disclosure is not limited to Figure 1 For example, in implementing the proposed method of the present disclosure, Figure 1 Some operations shown in the figure can be omitted, and Figure 1 Operations not shown in the figure may be included in the method. In addition, the execution order of some operations may be changed.

[0067] As described above, in the proposed method of the present disclosure, the output rate (or gain) of the speaker (or audio signal) is controlled and / or adjusted to output sound by positioning the position of an audio object related to a sound source or an image object (video object) at the position of the sound source or image object (video object) emitting the sound.

[0068] In S102, the device may receive a multimedia signal. For example, the multimedia (e.g., audiovisual) signal may include a broadcast signal or a signal transmitted through a YouTube live stream. The multimedia signal may include, for example, a video signal and an audio signal. In the present disclosure, a video signal may be used as a term referring to a compiled video bitstream or as a term referring to a signal obtained by decoding a compiled video bitstream. Similarly, in the present disclosure, an audio signal may be used as a term referring to a compiled audio bitstream or a term referring to a signal obtained by decoding a compiled audio bitstream.

[0069] In S104, the apparatus may obtain an audio frame and a video frame from a multimedia signal. In detail, the multimedia signal may be divided into an audio frame and a video frame. The audio frame and the video frame may be obtained by continuously separating an audio signal and a (still) image signal (or a video signal) from a multimedia (e.g., audiovisual) signal at predetermined time intervals. For example, an audio frame may refer to an array of audio samples corresponding to a specific time length, and a video frame may refer to a two-dimensional array of video samples at a specific time. For example, in S104, the apparatus may Figure 2 As shown in FIG. , audio frames and video frames are obtained from a multimedia signal.

[0070] In S106, the device may obtain at least one audio object from an audio frame, and obtain at least one video object from a video frame. In particular, the device may obtain (or separate) a single audio object or multiple audio objects (or at least one audio object) from an audio frame. The audio signal of an audio frame is a mixed signal, which is a mixture of signals generated by multiple different sound sources, signals generated from a single sound source, or a "silent" signal. An audio object is an audio signal separated and obtained from an audio frame. The device may obtain (or separate) a single video object or multiple video objects (or at least one video object) from a video frame. A video frame is a (still) image obtained at a specific time interval. The (still) image may contain people / animals and various object shapes. A video object is a (still) image block obtained by separating an area of ​​a (still) image of a video frame. For example, in S106, the device may Figure 3 As shown in FIG. 1 , at least one audio object is obtained from an audio frame. For example, in S106, the device may obtain at least one video object from a video frame, such as Figure 4 Additionally / alternatively, S106 may correspond to Figure 5 The audio separation 512 and video object detection 530 of the apparatus. In S106, the operation of the apparatus may include Figure 5 The audio separation 512 and video object detection 530 operations are described.

[0071] In S108, the device may perform video classification on at least one video object (obtained in S106) and perform audio classification on at least one audio object (obtained in S106). That is, in S108, the device classifies objects (e.g., audio objects and / or video objects) obtained from audio (or audio frames) and video (or video frames). Audio classification may include determining the type of sound source from which the audio object signal is generated. There is no limitation on the type of sound source classified by audio classification. Video classification may include determining the type of object represented by the video object signal (or (still) image block). There is no limitation on the type of object classified by video classification. For example, in S108, the device may perform audio classification on audio objects and perform video classification on video objects, such as Figure 6 Additionally / alternatively, S108 may correspond to Figure 5 The operation of the device in S108 may include the following: Figure 5 The operations described by the audio classification 518 and the video classification 532.

[0072] In S110, the apparatus may obtain feature information by performing feature extraction on at least one video object and at least one audio object (obtained in S106). A plurality of features may be extracted in S110. For example, S110 may correspond to Figure 5 Audio feature extraction 518 and video feature extraction 532. Reference Figure 5 , the operation of the apparatus in S110 may include the operations described with respect to audio feature extraction 518 and video feature extraction 532. Additionally / alternatively, the operation of the apparatus in S110 may include reference to Figure 7 Operations described and / or referenced Figure 8 Describes the operation.

[0073] In S112, the device may determine the correlation between at least one video object and at least one audio object based on the feature information (obtained in S110) and the object matching engine. Specifically, the device may input the features of the audio / video into the matching engine to determine the relationship therebetween. For example, in S112, the device may determine the correlation between at least one video object and at least one audio object, such as Fig. 9 Additionally / alternatively, S112 may correspond to Figure 5 The object matching engine 540, and the operation of the device in S112 may include Figure 5 The operation of the object matching engine 540 is described.

[0074] In S116, the device may perform directional rendering on the specific audio object based on the screen position of the video object related to the specific audio object according to the correlation (determined in S112). Directional rendering includes controlling and / or adjusting the output rate (or gain) of the speaker (or audio signal) according to the screen position of the video object when the object matching engine determines that the audio object sound source is the video object on the screen. For example, in S116, the device may perform directional rendering on the specific audio object based on the screen position of the video object related to the specific audio object, such as Fig.10 Additionally / alternatively, S116 may correspond to Figure 5 Directed rendering 516, and the operation of the device in S116 may include Figure 5 The operation of directed rendering 516 is described.

[0075] Figure 2 The figure shows an example of obtaining audio frames and video frames from a multimedia signal. Figure 1 The S104 can be executed Figure 2 Although Figure 2 The figure shows that the audio and video frames are obtained 25 times per second or at intervals of 40 ms, but this is for illustration purposes only. The proposed method is not limited to Figure 2 .

[0076] The audio frame and the video frame are obtained by continuously separating the audio signal and the (still) image signal from the multimedia (e.g., audio-visual) signal at predetermined time intervals. As the time interval between obtaining the audio frame and obtaining the video frame is shortened, the accuracy of determining the relationship between the audio object and the video object can be improved.

[0077] For example, refer to Figure 2 , when audio frames and video frames are obtained 25 times per second, that is, at intervals of 40 ms, an audio frame consisting of a 40 ms long audio signal and a video frame consisting of a (still) image are obtained every 40 ms.

[0078] The audio signal of the audio frame is a mixed signal, which is a mixture of signals generated by multiple different sound sources, signals generated from a single sound source, or a "silent" signal. The audio object is an audio signal separated and obtained from the audio frame. For example, when the audio frame is a signal generated from a single sound source, one audio object can be obtained. For example, when the audio frame is a mixed signal, multiple audio objects can be obtained from the sound sources respectively. The residual signal that cannot be separated for each sound source is processed on a separate immersive sound processing path (for example, see Figure 5514). A single audio object or multiple audio objects separated from the audio frame are used to analyze the relationship with the video object obtained from the video frame.

[0079] Figure 3 An example of obtaining at least one audio object from an audio frame is illustrated. Figure 3 The operations shown in the figure can be Figure 1 Executed in S106 of Figure 3 In the example of , it is assumed that an audio object (e.g., audio object [n, 1]) is obtained from an audio frame (e.g., audio frame [n]). However, the method proposed in the present disclosure is not limited thereto. The method can even be applied to the case where multiple audio objects are obtained from an audio frame.

[0080] For example, refer to Figure 3 , when a mixed signal of an audio frame (e.g., audio frame [n]) consists of a voice (voice) generated from one sound source (e.g., a person) and background noise, an audio object (e.g., audio object [n, 1]) can be obtained by separating the human voice, and the background noise can be separated into a residual signal. Audio frame [n] indicates the nth audio frame, and audio object [n, 1] indicates an audio object obtained from the nth audio frame.

[0081] A video frame is a (still) image obtained at a specific time interval, and the (still) image contains people / animals and various other types of object shapes. A video object is a (still) image block obtained by separating an area of ​​the (still) image of a video frame. A video object is a block obtained by separating only the area of ​​the object from the entire (still) image area, so that people / animals and various types of objects can be clearly distinguished. When the image blocks of the video object are separated, the object can also be classified. For example, according to the proposed method of the present disclosure, a video object of a person / animal / vehicle can be obtained, which can be a sound source. A video object of a person may include a still image block of a face or lip part that produces a voice.

[0082] Figure 4 An example of obtaining at least one video object from a video frame is shown. Figure 1 Executed in S106 Figure 4 The operation shown in the figure. Figure 4 In the embodiment, it is assumed that three video objects (e.g., video object [n, 1], video object [n, 2], and video object [n, 3]) are obtained from one video frame (e.g., video frame [n]). However, the proposed method of the present disclosure is not limited thereto and can even be applied to a case where a different number of video objects than three video objects are obtained from one video frame.

[0083] A single video object or a plurality of video objects separated from a video frame is used to analyze a relationship between the video object and an audio object obtained from an audio frame.

[0084] For example, refer to Figure 4 , in an n-th still image (e.g., video frame [n]), image blocks of a face region represented by three rectangular regions are separated, and then video object [n, 1], video object [n, 2], and video object [n, 3] are obtained from left to right. Video frame [n] represents an n-th video frame, and video object [n, 1], video object [n, 2], and video object [n, 3] represent a first video object, a second video object, and a third video object respectively obtained from the n-th frame.

[0085] Figure 5 A block diagram illustrating sound object following according to the method proposed in the present disclosure.

[0086] exist Figure 5 In the example of , the audio separation 512 includes an operation of separating (or demultiplexing) (or de-multiplexing) the audio input signal 502 into an audio object and a residual signal (or background signal).

[0087] exist Figure 5 In the example of , immersive surround is performed on a separated background signal or audio signal determined to have no relationship between audio and video by an object matching engine 514. Immersive surround can include processing of traditional stereo and surround sound.

[0088] exist Figure 5 In an example of , directional rendering 516 may include processing performed to output an audio signal (or a separated audio object) at a specific screen position (by positioning a sound image of the audio signal (or separated audio object)) based on a correlation between the audio signal (or a separated audio object) and an image object (video object). The directional rendering 516 controls and / or adjusts an output rate (or gain) of a speaker (or an audio signal) based on the screen position to positionally position the sound image of the audio signal. More specifically, the directional rendering 516 may include operations to control and / or adjust an output rate (or gain) of a speaker (or an audio signal) through which an audio object is output based on a position on the screen of an image object related to the audio signal (or a separated audio object). For example, the directional rendering 516 may correspond to Figure 1 S114.

[0089] exist Figure 5In the example of , audio classification 518 is performed to determine the type of sound source from which the audio object signal is generated. There is no limitation on the type of sound source classified by audio classification 518. The results of audio classification 518 of audio objects can be used as input features of matching engine 540 to determine the relationship between audio objects and video objects. For example, audio classification 518 can correspond to Figure 1 S108.

[0090] For example, Figure 6 As shown in , audio classification 518 can classify audio object signals into four different types, such as human voice, animal sound, vehicle sound and others. When the type of sound source from which the audio object signal is generated cannot be determined, the signal can be classified as "others".

[0091] Video object detection 530 may be performed to detect at least one video object from the video input signal 504. As described above, a video object may include a (still) image block contained in a video frame. For example, video object detection 530 may correspond to Figure 1 S106.

[0092] Video classification 532 is performed to determine the type of object corresponding to the video object (still) image block. There is no limitation on the type of object classified by video classification 532. The results of video classification 532 of video objects can be used as input features of matching engine 540 to determine the relationship between audio objects and video objects. For example, video classification 532 can correspond to Figure 1 S108.

[0093] For example, Figure 6 As shown in FIG. 5 , the video classification 532 can classify the video object into four types, such as human face, animal, vehicle and others. When the type of the video object cannot be determined, the video object can be classified into others.

[0094] exist Figure 5 In the example of , the object matching engine 540 is a model learned using deep learning (DL), machine learning (ML), reinforcement learning, etc. based on the feature values ​​of audio and video as input values. For example, the object matching engine may include a model that has been trained based on a large amount of data and ANN through learning such as DL, ML, or reinforcement learning. For example, the object matching engine 540 may correspond to Figure 1 S112.

[0095] The process of extracting useful components from audio and video is called feature extraction. Figure 5In the example of FIG. 5 , feature extraction 518 , 532 is performed on at least one audio object and at least one video object to obtain feature values. The feature values ​​may be referred to as feature information. For example, feature extraction 518 , 532 may correspond to Figure 1 S110.

[0096] In audio feature extraction 518, various methods conventionally used in speech recognition or audio recognition may be used to obtain or extract audio feature values. Additionally / alternatively, the audio feature values ​​may include audio start information for determining whether a voice has been uttered.

[0097] Representative examples of traditional speech recognition technologies include LPC, Log-mel, and MFCC. LPC stands for linear prediction coding, which is a speech generation model that uses a specific periodic pulse sequence and Gaussian white noise as excitation sources to generate speech through an LPC filter.

[0098] Log-Mel is a unit that represents the nonlinear frequency characteristics of the human ear. That is, humans have high resolution in the low frequency region and low resolution in the high frequency region, which exhibits logarithmic characteristics over the frequency band. The magnitude or power of the spectrum can be estimated by performing Fourier transform (e.g., Fast Fourier Transform (FFT)) on samples of the audio portion, and then used to obtain the log-Mel through a filter bank configured in a Mel scale.

[0099] MFCC stands for Mel-Frequency Cepstral Coefficients. MFCC can be obtained using a Cepstral Transform, which removes correlations in the log-Mel spectrum.

[0100] In video feature extraction 532, various methods commonly used for video object recognition may be used to obtain or extract video feature values. Additionally / alternatively, a face box, a lip contour, and / or a comparison value of its value may be used to obtain or extract video feature values.

[0101] For example, Figure 7 The figure shows an example of obtaining or extracting video feature values ​​based on face frame and lip contour. Figure 7As illustrated in , it is determined whether a person's face is recognized, and the height H and horizontal length W of the face frame are determined. The face frame refers to a rectangular area determined based on face recognition, the height of the face frame refers to the height of this rectangular area, and the horizontal length of the face frame refers to the width of this rectangular area. H and W can be used to normalize the vertical and horizontal values ​​of the lips (for example, see Formula 1). The vertical and horizontal lengths of the lips are determined using information obtained from the lip contour. By comparing the vertical and horizontal lengths of the lips according to the video frame, it can be determined whether the lips are moving. By comparing the current frame with the previous frame regarding the movement of the lips, information about the amount of movement of the lips can be obtained.

[0102] More specifically, in Figure 7 In the example of , the video feature value can be obtained or extracted based on Equation 1. In Equation 1, H represents the height of the face frame, W represents the width of the face frame, D1 represents the vertical length of the lip contour, D2 represents the horizontal length of the lip contour, d1 represents the normalized value of the vertical length of the lip contour based on the height H of the face frame, d2 represents the normalized value of the horizontal length of the lip contour based on the width W of the face frame, n represents the frame number, and M1 represents the difference in the vertical length of the lip contour between the current frame n and the previous frame n-1, M2 represents the difference in the vertical length of the lip contour between the current frame n and the previous frame n-1, A1 represents the difference between the M1 value of the current frame n and the M1 value of the previous frame n-1, and A2 represents the difference between the M2 value of the current frame n and the value M2 in the previous frame n-1.

[0103] [Equation 1]

[0104]

[0105] In Equation 1, M1 represents a comparison result between vertical lengths of lips in a video frame, M2 represents a comparison result between horizontal lengths of lips in a video frame, and A1 and A2 represent information about a movement amount of lips determined by comparing the movement of lips with a previous frame. In Equation 1, at least one of M1, M2, A1, and A2 may be used as a video feature value.

[0106] As another example, refer to Figure 8 , the lip position ratio (e.g., A:B:C, D:E:F) in the face frame may be used as a video feature value. Specifically, when the face frame is referred to as a first rectangular area and the rectangular area containing the lip contour is referred to as a second rectangular area, the video feature value may include a distance between a top boundary of the second rectangular area and a top boundary of the first rectangular area in the first rectangular area (e.g., Figure 8 A in), the height of the second rectangular area (for example, Figure 8 B in the figure), the distance between the bottom boundary of the second rectangular area and the bottom boundary of the first rectangular area (for example, Figure 8and C) in the second rectangular area (for example, A:B:C); and the distance between the left boundary of the second rectangular area and the left boundary of the first rectangular area (for example, Figure 8 D in), the height of the second rectangular area (for example, Figure 8 E in the figure), and the distance between the right border of the second rectangular area and the right border of the first rectangular area (for example, Figure 8 In some embodiments, the video feature value may be obtained by normalizing the lip position ratio (e.g., A:B:C, D:E:F) between the lip positions (e.g., A:B:C, D:E:F), or the video feature value may be obtained based on these ratios. For example, the video feature value may be obtained or extracted by normalizing the lip position ratio (e.g., A:B:C, D:E:F).

[0107] In the proposed method of the present disclosure, reference is made to Figure 7 Describes the video feature values ​​and references Figure 8 The video feature values ​​described can be used independently or in combination. Figure 7 Describes the video feature values ​​and references Figure 8 At least one of the described video feature values ​​may be used as input to the object matching engine 540 .

[0108] Return to reference Figure 5 , when feature values ​​of an audio signal (or audio object) and a video signal (or video object) obtained or extracted from a multimedia signal such as a broadcast or YouTube are input to an object matching engine 540 that is a model trained using the audio and video feature values ​​as input values, the degree of relationship between the audio signal (or audio object) and the video signal (or video object) can be obtained or determined. For example, the degree of relationship between the audio object and the video object can be expressed as a (real number) value, which can be called a ratio, a correlation, information indicating the degree of relationship, information about the relationship, etc. The object matching engine 540 can perform an operation of determining the degree of relationship between audio / video objects (or obtaining information indicating the degree of relationship) based on the results of the classification 518, 532 of the audio / video objects and / or the results of the feature extraction 518 / 532 of the audio / video objects (using a model that has been trained by DL, ML, reinforcement learning, etc.).

[0109] Fig. 9 An example of the operation of the object matching engine 540 is shown. Fig. 9 In the example, it is assumed that three video objects (e.g., video 1, video 2, video 3) are obtained from one video frame, and one audio object (e.g., audio 1) is obtained from one audio frame. However, the method is not limited thereto, and can even be applied to a case where the number of obtained video objects and / or audio objects is different from the assumption.

[0110] refer to Fig. 9, when three video objects (e.g., video 1, video 2, video 3) are obtained from a video (or video frame) on one screen, these objects are classified into corresponding categories. Video 1 and video 3 are classified into the human category, and video 2 is classified into the animal category. Audio 1 is an audio signal (or audio object) obtained by removing background sound from the original sound. When feature values ​​are extracted from video objects (e.g., video 1, video 2, video 3) and audio objects (e.g., audio 1) and provided as input to the object matching engine 540, the object matching engine 540 outputs the degree of relationship as a ratio. Fig. 9 In the example of , a higher ratio may mean a higher degree of relationship between video and audio. Alternatively, it may be configured so that a lower ratio means a higher degree of relationship between video and audio.

[0111] For example, based on the value (e.g., ratio) of the information indicating the degree of relationship (or the information about the relationship) being greater than a predetermined value, the video object related to the value (e.g., ratio) of the information indicating the degree of relationship (or the information about the relationship) may be determined to be a video object related to the audio object (e.g., audio 1). Additionally / alternatively, when Fig. 9 When there are multiple video objects as in the example, the video object (video 1) associated with the maximum value (e.g., 0.9) of the information indicating the degree of relationship (or information about the relationship) (e.g., a ratio) can be determined as a video object related to the audio object (e.g., audio 1).

[0112] As another example, based on the value (e.g., ratio) of the information indicating the degree of relationship (or the information about the relationship) being less than a predetermined value, the video object related to the value (e.g., ratio) of the information indicating the degree of relationship (or the information about the relationship) may be determined to be a video object related to the audio object (e.g., audio 1). Additionally / alternatively, when Fig. 9 When there are multiple video objects as in the example, the video object (video 3) related to the minimum value (e.g., 0.1) of the information indicating the degree of relationship (or information about the relationship) (e.g., a ratio) can be determined as the video object related to the audio object (e.g., audio 1).

[0113] Return to reference Figure 5 , directional rendering 516 may include controlling and / or adjusting the output rate (or gain) of a speaker (or audio signal) according to the screen position of a video object when the object matching engine 540 determines that the audio object sound source or sound generating object is a video object on the screen. Figure 4For example, when the object matching engine 540 determines that the video object [n, 3] is related to the audio object, the directional rendering 516 can be performed to control and / or adjust the output rate (or gain) of the speaker (or audio signal) according to the screen position of the video object [n, 3].

[0114] The output rate (or gain) of each speaker is controlled and / or adjusted for sound image localization at the screen position of the video object relative to the user. The number of speakers whose output rates (or gains) are controlled and / or adjusted is greater than or equal to 2. Depending on the size of the device, the position of the user, and the number and arrangement of the speakers, the output rate of the sound image localization may vary.

[0115] Fig.10 An example of sound image localization performed by a device including four speakers according to the screen position of a video object is illustrated. Fig.10 It is only an example, and the proposed method of the present disclosure may be applied to devices including a different number of speakers in the same manner / similar manner.

[0116] For example, refer to Fig.10 , when four speakers are arranged on the upper left LT / upper right RT / lower left LB / lower right RB of the TV, sound image positioning is performed at a position of coordinates (X, Y) relative to the screen center coordinates (e.g., (0,0)) for the screen width (e.g., screen_width) and the screen height (e.g., screen_height), the output rate (or gain) GLT / GRT / GLB / GRB for the speaker positions LT / RT / LB / RB can be determined as follows. Here, LT means upper left, RT means upper right, LB means lower left, RB means lower right, GLT means the gain of the upper left, GRT means the gain of the upper right, GLB means the gain of the lower left, and GRB means the gain of the lower right.

[0117] Based on the “distance between the user and the screen (e.g. Fig.10 User_Screen_Distance in ), horizontal speaker spacing (e.g. Fig.10 Horizontal_Speaker_Spacing in ), screen width (e.g. Fig.10 screen_width in ), screen_height (for example, Fig.10 The horizontal output rate for coordinate X and the vertical output rate for coordinate Y are determined based on the "Screen_Height in (screen_height)", and GLT, GRT, GLB, and GRB are determined so that GLT+GRT+GLB+GRB is equal to a constant value based on the horizontal output rate and the vertical output rate.

[0118] Fig.11 and Fig.12An example of the operation of sound object following when there are two or four speakers is illustrated respectively. SCF represents a scene change flag, and GL and GR represent the gain of the left audio and the gain of the right audio. The audio starts to represent the probability of the sound output of the audio object. For example, for human speech, whether a human is speaking is indicated by a probability between 0 (not speaking) and 1 (speaking). Detect a face, and determine the position of the face by analyzing the correlation between the detected face and the speech. Then, control and / or adjust the left and right voice signal output gains so that the speech is output at the position where the face is located. The sum of the output rates of the outputs from all speakers should be 1. By increasing the output of the speaker with a higher correlation between audio and video, an improved 3D effect and an improved sense of immersion can be provided for the sound. The output rate can be adjusted according to the display size of the device, the number and arrangement of the speakers of the device, and the sound characteristics.

[0119] Return to reference Figure 5 , mixing 520 may be performed to mix the signal processed in immersive surround 514 with the signal processed in directional rendering 516 to generate an audio output signal 506 .

[0120] Device structure

[0121] Fig.13 The block diagram of the signal processing device of the present disclosure is illustrated. The device 10 according to the present invention can be configured to process a signal including a video signal and an audio signal according to the present invention. For example, the signal processing device 10 to which the present invention can be applied can include a mobile terminal such as a smart phone, a portable device such as a laptop computer, a household appliance such as a digital TV and a digital video player, and a transportation tool such as a car. As another example, the signal processing device 10 to which the present invention can be applied can be included as a part of an application-specific integrated circuit (ASIC) implemented in the form of a system on a chip (SoC).

[0122] The memory 12 may store programs for processing and control of the processor 11, and may store bit streams including video signals and audio signals, decoded video signals, decoded audio signals, control information required for signal processing, temporary data generated during signal processing, etc. In addition, the memory 12 may be used as a buffer for various video and audio signals. The memory 12 may be implemented as a storage device such as a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a static RAM (SRAM), a hard disk drive (HDD), a solid-state drive (SSD), etc.

[0123] The processor 11 controls the operation of each module in the signal processing device. In particular, the processor 11 can perform various control functions to perform the signal processing method according to the present invention. The processor 11 can also be referred to as a controller, a microcontroller, a microprocessor, a microcomputer, etc. The method according to the present invention can be implemented by hardware, firmware, software, or a combination thereof. When hardware is used to implement the present invention, a dedicated integrated circuit (ASIC) or a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), and a field programmable gate array (FPGA) configured to perform the present invention can be set in the processor 11. In the implementation of the method according to the present invention using firmware or software, the firmware or software can be configured to include a module, a process, or a function for performing the function or operation of the present invention. The firmware or software configured to perform the method according to the present invention can be stored in the memory 12, or can be stored in a computer-readable recording medium (not shown) separated from the memory 12, and when executed by the processor 11, the device 10 performs the method according to the present invention.

[0124] In addition, the device 10 may optionally include a network interface module (NIM) 13. The NIM 13 may be operably connected to the processor 11, and the processor 11 may control the NIM 13 to send or receive wireless / wired signals for carrying information and / or data, signals and messages through a wireless / wired network. The NIM 13 may support various communication standards, such as, for example, IEEE 802 series, 3GPP LTE (-A), 3GPP 5G, Wi-Fi, Advanced Television Systems Committee (ATSC) and Digital Video Broadcasting (DVB). According to the communication standard, video and audio signals such as control information and / or encoded bit streams may be sent and received. The NIM 13 may not be included in the device as needed.

[0125] In addition, the device 10 may optionally include an input / output interface 14. The input / output interface 14 is operably connected to the processor 11, and the processor 11 may control the input / output interface 14 to receive or output control signals and / or data signals. The input / output module 14 may support standards such as, for example, a universal serial bus (USB), Bluetooth, near field communication (NFC), a serial / parallel interface, a digital visual interface (DVI), and a high-definition multimedia interface (HDMI) so as to be connected to input devices such as a keyboard, a mouse, a touch pad, or a camera, and output devices such as a display.

[0126] Fig.14 A more detailed block diagram of an apparatus to which the present invention may be applied is shown.

[0127] The device 100 may include a memory 130 and a controller (or processor) 120 , and may optionally further include a wireless communication device 110 , an input device 150 , a sensing device 160 , an output device 170 , an interface controller 180 , and a power supply 190 . Fig.14 The components shown in are not necessary to implement the device 100. Therefore, a device applying the methods described herein may have more or fewer components than those listed above.

[0128] More specifically, the wireless communication device 110 of the components may include one or more modules that enable wireless communication between the device 100 and a wireless communication system, between the device 100 and another device 100, or between the device 100 and an external server. In addition, the wireless communication device 110 may include one or more modules for connecting the device 100 to one or more networks.

[0129] The wireless communication device 110 may include at least one of a broadcast receiving module 111 , a mobile communication module 112 , a wireless Internet module 113 , a near field communication (NFC) module 114 , and a positioning information module 115 .

[0130] The input device 150 may include a camera 151 or an image input device configured to input an image signal, a microphone 152 or an audio input device configured to input an audio signal, or a user input device 153 (e.g., a touch key, a mechanical key, etc.) configured to receive information from a user. The audio data or image data collected by the input device 150 may be analyzed and processed into a user's control command.

[0131] The sensing device 160 may include one or more sensors configured to sense at least one of information within the device, information about the surrounding environment around the device, and user information. For example, the sensing device 160 may include a proximity sensor 161, an illumination sensor 162, a touch sensor, an acceleration sensor, a magnetic sensor, a G sensor, a gyroscope sensor, a motion sensor, an RGB sensor, an infrared (IR) sensor, a finger scan sensor, an ultrasonic sensor, an optical sensor (e.g., see camera 151), a microphone 152, a battery gauge, an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a thermal sensor, a gas sensor, etc.), and a chemical sensor (e.g., an electronic nose, a health sensor, a biometric sensor, etc.). The device disclosed herein may combine and utilize information sensed by at least two or more of these sensors.

[0132] The output device 170 for generating output related to vision, hearing or touch may include at least one of a display 171, a sound output device 172, a haptic module 173 and an optical output device 174. The display 171 may form a layer structure with a touch sensor or be integrated with the touch sensor to implement a touch screen. The touch screen may be used as a user input device 153 that provides an input interface between the device 100 and the user, and may also provide an output interface between the device 100 and the user.

[0133] The interface controller 180 serves as a path to various types of external devices connected to the device 100. The interface controller 180 may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port configured to connect a device equipped with an identification module, an audio input / output (I / O) port, a video input / output (I / O) port, and a headset port. In response to the connection of an external device to the interface controller 180, the device 100 may perform appropriate control related to the connected external device.

[0134] In addition, the memory 130 stores data supporting various functions of the device 100. The memory 130 can store a plurality of applications (or applications) driven by the device 100, data for the operation of the device 100, and instructions. At least some of these applications can be downloaded from an external server via wireless communication. In addition, at least some of these applications can exist on the device 100 from the time of shipment to enable the basic functions of the device 100 (e.g., call reception, call transmission, message reception, and message transmission). The application can be stored in the memory 130 installed on the device 100 and driven by the controller 120 to perform the operation (or function) of the device 100.

[0135] In addition to the operations related to the application, the controller 120 generally controls the overall operation of the device 100. The controller 120 may provide appropriate information or functions to the user, or may process signals, data, information, etc. input or output through the above components or by driving the application stored in the memory 130.

[0136] In addition, the controller 120 may control the reference Fig.14 At least some of the described components are in order to drive the application program stored in the memory 130. In addition, the controller 120 may combine and operate at least two components included in the device 100 to drive the application program.

[0137] Under the control of the controller 120, the power supplier 190 is supplied with power from an external power source or an internal power source to supply power to each component included in the device 100. The power supplier 190 may include a battery, which may be a built-in battery or a replaceable battery.

[0138] At least some of the above components can operate in cooperation with each other to implement the operation, control or control method of the device according to the present invention. In addition, the operation, control or control method of the device 100 can be implemented on the device 100 by driving at least one application stored in the memory 130.

[0139] Examples of wireless devices applicable to the present disclosure

[0140] Fig.15 Another example of a wireless device applied to the present disclosure is illustrated. The wireless device may be implemented in various forms depending on use cases / services.

[0141] refer to Fig.15 , the wireless device 100 may be configured by various elements, components, units / parts and / or modules. For example, each of the wireless devices 100 and 200 may include a communication unit 110, a control unit 120, a memory unit 130 and an additional component 140. The communication unit may include a communication circuit and a transceiver. For example, the communication circuit may include one or more processors and / or one or more memories. For example, the transceiver may include one or more transceivers and / or one or more antennas. The control unit 120 is electrically connected to the communication unit 110, the memory 130 and the additional component 140, and controls the overall operation of the wireless device. For example, the control unit 120 may control the electrical / mechanical operation of the wireless device based on the program / code / command / information stored in the storage unit 130. The control unit 120 may send the information stored in the storage unit 130 to the outside (e.g., other communication devices) via the communication unit 110 through a wireless / wired interface, or store the information received from the outside (e.g., other communication devices) via the communication unit 110 through a wireless / wired interface in the storage unit 130.

[0142] Additional components 140 may be configured differently depending on the type of wireless device. For example, additional components 140 may include at least one of a power supply unit / battery, an input / output (I / O) unit, a drive unit, and a computing unit. The wireless device may be implemented in the form of, but not limited to, a robot, a vehicle, an XR device, a handheld device, a home appliance, an IoT device, a digital broadcast terminal, a holographic device, a public safety device, an MTC device, a medical device, a FinTech device (or a financial device), a security device, a climate / environmental device, an AI server / device, a base station, a network node, etc. The wireless device may be used in a mobile or fixed location depending on the use case / service.

[0143] exist Fig.15 In the wireless device 100, the various elements, components, units / parts and / or modules in the wireless device 100 may be connected to each other as a whole through a wired interface, or at least a part thereof may be wirelessly connected through the communication unit 110. For example, in each wireless device 100, the control unit 120 and the communication unit 110 may be connected by wire, and the control unit 120 and the first unit (e.g., 130 and 140) may be wirelessly connected through the communication unit 110. Each element, component, unit / part and / or module within the wireless device 100 may further include one or more elements. For example, the control unit 120 may be configured by a collection of one or more processors. As an example, the control unit 120 may be configured by a collection of a communication control processor, an application processor, an electronic control unit (ECU), a graphics processing unit and a memory control processor. As another example, the memory 130 may be configured by a collection of a random access memory (RAM), a dynamic RAM (DRAM), a read-only memory (ROM), a flash memory, a volatile memory, a non-volatile memory and / or a combination thereof.

[0144] Hereinafter, the implementation will be described in detail with reference to the accompanying drawings. Fig.15 .

[0145] Examples of Handheld Devices Applicable to the Present Disclosure

[0146] Fig.16 The diagram shows a handheld device applied to the present disclosure. The handheld device may include a smart phone, a smart tablet, a wearable device (e.g., a smart watch or smart glasses), or a portable computer (e.g., a laptop). The handheld device may be referred to as a mobile station (MS), a user terminal (UT), a mobile subscriber station (MSS), a subscriber station (SS), an advanced mobile station (AMS), or a wireless device terminal (WT).

[0147] refer to Fig.16 , the handheld device 100 may include an antenna unit 108, a communication unit 110, a control unit 120, a storage unit 130, a power supply unit 140a, an interface unit 140b, and an I / O unit 140c. The antenna unit 108 may be configured as a part of the communication unit 110. Blocks 110 to 130 / 140a to 140c correspond to Fig.16 Blocks 110 to 130 / 140.

[0148] The communication unit 110 may send signals (e.g., data and control signals) to other wireless devices or BSs, and receive signals (e.g., data and control signals) from other wireless devices or BSs. The control unit 120 may perform various operations by controlling the constituent elements of the handheld device 100. The control unit 120 may include an application processor (AP). The storage unit 130 may store data / parameters / programs / codes / commands required to drive the handheld device 100. The storage unit 130 may store input / output data / information. The power supply unit 140a may supply power to the handheld device 100, and may include a wired / wireless charging circuit, a battery, etc. The interface unit 140b may support the connection of the handheld device 100 with other external devices. The interface unit 140b may include various ports (e.g., audio I / O ports and video I / O ports) for connecting to external devices. The I / O unit 140c may input or output video information / signals, audio information / signals, data and / or information input by a user. The I / O unit 140c may include a camera, a microphone, a user input unit, a display unit 140d, a speaker, and / or a haptic module.

[0149] As an example, in the case of data communication, the I / O unit 140c may acquire information / signals (e.g., touch, text, voice, image, or video) input by a user, and the acquired information / signals may be stored in the storage unit 130. The communication unit 110 may convert the information / signals stored in the memory into radio signals, and directly transmit the converted radio signals to other wireless devices or to a BS. The communication unit 110 may receive radio signals from other wireless devices or BSs, and then restore the received radio signals to the original information / signals. The restored information / signals may be stored in the storage unit 130, and may be output as various types (e.g., text, voice, image, video, or tactile) through the I / O unit 140c.

[0150] The vehicle or autonomous vehicle used in the present invention driving vehicle Example

[0151] Fig.17 The diagram is applied to a vehicle or an autonomous vehicle of the present disclosure. The vehicle or the autonomous vehicle can be implemented by a mobile robot, a car, a train, a manned / unmanned aerial vehicle (AV), a ship, etc.

[0152] refer to Fig.17, the vehicle or autonomous driving vehicle 100 may include an antenna unit 108, a communication unit 110, a control unit 120, a drive unit 140a, a power supply unit 140b, a sensor unit 140c and an autonomous driving unit 140d. The antenna unit 108 may be configured as a part of the communication unit 110. Blocks 110 / 130 / 140a to 140d correspond to Fig.15 Block 110 / 130 / 140.

[0153] The communication unit 110 may send signals (e.g., data and control signals) to and receive signals (e.g., data and control signals) from external devices such as other vehicles, BSs (e.g., gNBs and roadside units), and servers. The control unit 120 may perform various operations by controlling elements of the vehicle or autonomous driving vehicle 100. The control unit 120 may include an electronic control unit (ECU). The driving unit 140a may enable the vehicle or autonomous driving vehicle 100 to travel on a road. The driving unit 140a may include an engine, a motor, a transmission system, wheels, brakes, a steering device, and the like. The power supply unit 140b may supply power to the vehicle or autonomous driving vehicle 100, and may include a wired / wireless charging circuit, a battery, and the like. The sensor unit 140c may acquire vehicle status, surrounding environment information, user information, and the like. The sensor unit 140c may include an inertial measurement unit (IMU) sensor, a collision sensor, a wheel sensor, a speed sensor, a slope sensor, a weight sensor, a heading sensor, a location module, a vehicle forward / backward sensor, a battery sensor, a fuel sensor, a tire sensor, a steering sensor, a temperature sensor, a humidity sensor, an ultrasonic sensor, an illumination sensor, a pedal position sensor, etc. The autonomous driving unit 140d may implement a technology for maintaining a lane on which the vehicle is traveling, a technology for automatically adjusting a speed such as an adaptive cruise control, a technology for autonomous driving along a determined path, a technology for automatically setting a path with a destination set, etc.

[0154] For example, the communication unit 110 can receive map data, traffic information data, etc. from an external server. The autonomous driving unit 140d can generate an autonomous driving path and a driving plan from the obtained data. The control unit 120 can control the drive unit 140a so that the vehicle or the autonomous driving vehicle 100 can move along the autonomous driving path according to the driving plan (e.g., speed / direction control). In the middle of autonomous driving, the communication unit 110 can obtain the latest traffic information data from the external server irregularly / regularly, and can obtain the surrounding traffic information data from the neighboring vehicles. In the middle of autonomous driving, the sensor unit 140c can obtain the vehicle state and / or surrounding environment information. The autonomous driving unit 140d can update the autonomous driving path and driving plan based on the newly obtained data / information. The communication unit 110 can transmit information about the vehicle position, autonomous driving path and / or driving plan to the external server. The external server can predict the traffic information data based on the information collected from the vehicle or the autonomous driving vehicle, using AI technology, etc., and provide the predicted traffic information data to the vehicle or the autonomous driving vehicle.

[0155] Examples of vehicles suitable for the present disclosure

[0156] Fig.18 The illustration shows a vehicle applied to the present disclosure, which can be implemented as a transportation vehicle, an aircraft, a ship, etc.

[0157] refer to Fig.18 , the vehicle 100 may include a communication unit 110, a control unit 120, a storage unit 130, an I / O unit 140a, and a positioning unit 140b. Here, blocks 110 to 130 / 140a and 140b correspond to Fig.15 Blocks 110 to 130 / 140.

[0158] The communication unit 110 may send and receive signals (e.g., data and control signals) to external devices such as other vehicles or BSs. The control unit 120 may perform various operations by controlling the constituent elements of the vehicle 100. The storage unit 130 may store data / parameters / programs / codes / commands for supporting various functions of the vehicle 100. The I / O unit 140a may output AR / VR objects based on the information in the storage unit 130. The I / O unit 140a may include a HUD. The positioning unit 140b may obtain information about the location of the vehicle 100. The location information may include information about the absolute position of the vehicle 100, information about the position of the vehicle 100 in the driving lane, acceleration information, and information about the position of the vehicle 100 from the neighboring vehicles. The positioning unit 140b may include a GPS and various sensors.

[0159] As an example, the communication unit 110 of the vehicle 100 receives map information and traffic information from an external server and stores the received information in the storage unit 130. The positioning unit 140b can obtain the vehicle location information through GPS and various sensors and store the obtained information in the storage unit 130. The control unit 120 can generate a virtual object based on the map information, traffic information and vehicle location information, and the I / O unit 140a can display the generated virtual object in the window (1410 and 1420) in the vehicle. The control unit 120 can determine whether the vehicle 100 is driving normally in the driving lane based on the vehicle location information. If the vehicle 100 exits the driving lane abnormally, the control unit 120 can display a warning on the window of the vehicle through the I / O unit 140a. In addition, the control unit 120 can broadcast a warning message about driving abnormality to neighboring vehicles through the communication unit 110. Depending on the situation, the control unit 120 can send the vehicle location information and the information about driving / vehicle abnormality to the relevant organization.

[0160] Examples of XR Devices Suitable for the Present Disclosure

[0161] Fig.19 The diagram is applied to the XR device of the present disclosure. The XR device can be implemented by an HMD, a HUD installed in a vehicle, a television, a smartphone, a computer, a wearable device, a home appliance, a digital signage, a vehicle, a robot, etc.

[0162] refer to Fig.19 , the XR device 100a may include a communication unit 110, a control unit 120, a storage unit 130, an I / O unit 140a, a sensor unit 140b, and a power supply unit 140c. Here, blocks 110 to 130 / 140a to 140c correspond to Fig.15 Blocks 110 to 130 / 140.

[0163] The communication unit 110 may send signals (e.g., media data and control signals) to and receive signals (e.g., media data and control signals) from external devices such as other wireless devices, handheld devices, or media servers. The media data may include video, images, and sounds. The control unit 120 may perform various operations by controlling the constituent elements of the XR device 100a. For example, the control unit 120 may be configured to control and / or perform processes such as video / image acquisition, (video / image) encoding, and metadata generation and processing. The storage unit 130 may store data / parameters / programs / codes / commands required to drive the XR device 100a / generate an XR object. The I / O unit 140a may obtain control information and data from the outside and output the generated XR object. The I / O unit 140a may include a camera, a microphone, a user input unit, a display unit, a speaker, and / or a tactile module. The sensor unit 140b may obtain XR device status, surrounding environment information, user information, and the like. The sensor unit 140b may include a proximity sensor, an illumination sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, and / or a radar. The power supply unit 140c may supply power to the XR device 100a and may include a wired / wireless charging circuit, a battery, and the like.

[0164] For example, the storage unit 130 of the XR device 100a may include information (e.g., data) required to generate an XR object (e.g., an AR / VR / MR object). The I / O unit 140a may receive a command for manipulating the XR device 100a from a user, and the control unit 120 may drive the XR device 100a according to the user's drive command. For example, when a user desires to watch a movie or news through the XR device 100a, the control unit 120 sends content request information to another device (e.g., a handheld device 100b) or a media server through the communication unit 130. The communication unit 130 may download / stream content such as a movie or news from another device (e.g., a handheld device 100b) or a media server to the storage unit 130. The control unit 120 may control and / or perform processes such as video / image acquisition, (video / image) encoding, and metadata generation / processing for the content, and generate / output an XR object based on information about the surrounding space or real objects obtained through the I / O unit 140a / sensor unit 140b.

[0165] The XR device 100a may be wirelessly connected to the handheld device 100b through the communication unit 110, and the operation of the XR device 100a may be controlled by the handheld device 100b. For example, the handheld device 100b may be used as a controller of the XR device 100a. To this end, the XR device 100a may obtain information about the 3D position of the handheld device 100b, and generate and output an XR object corresponding to the handheld device 100b.

[0166] Examples of AI devices applicable to the present disclosure

[0167] Fig. 20 The AI ​​device applied to the present invention is shown in the figure. The AI ​​device can be implemented by a fixed device or a mobile device, such as a TV, a projector, a smart phone, a PC, a laptop, a digital broadcast terminal, a tablet computer, a wearable device, a set-top box (STB), a radio, a washing machine, a refrigerator, a digital signage, a robot, a vehicle, etc.

[0168] refer to Fig. 20 , the AI ​​device 100 may include a communication unit 110, a control unit 120, a memory unit 130, an I / O unit 140a / 140b, a learning processor unit 140c, and a sensor unit 140d. Blocks 110 to 130 / 140a to 140d correspond to Fig.15 Blocks 110 to 130 / 140.

[0169] The communication unit 110 may use wired / wireless communication technology to send wired / radio signals (e.g., sensor information, user input, learning model, or control signal) to and receive wired / radio signals (e.g., sensor information, user input, learning model, or control signal) from external devices such as other AI devices or AI servers. To this end, the communication unit 110 may send information in the storage unit 130 to the external device and send signals received from the external device to the storage unit 130.

[0170] The control unit 120 may determine at least one feasible operation of the AI ​​device 100 based on information determined or generated using a data analysis algorithm or a machine learning algorithm. The control unit 120 may perform an operation determined by controlling the constituent elements of the AI ​​device 100. For example, the control unit 120 may request, search, receive, or use data of the learning processor unit 140c or the storage unit 130, and control the constituent elements of the AI ​​device 100 to perform a predicted operation or an operation determined to be preferred among at least one feasible operation. The control unit 120 may collect historical information including the operation content of the AI ​​device 100 and the user's operation feedback, and store the collected information in the storage unit 130 or the learning processor unit 140c, or send the collected information to an external device such as an AI server. The collected historical information may be used to update the learning model.

[0171] The storage unit 130 may store data for supporting various functions of the AI ​​device 100. For example, the storage unit 130 may store data obtained from the input unit 140a, data obtained from the communication unit 110, output data of the learning processor unit 140c, and data obtained from the sensor unit 140. The storage unit 130 may store control information and / or software codes required to operate / drive the control unit 120.

[0172] The input unit 140a can obtain various types of data from the outside of the AI ​​device 100. For example, the input unit 140a can obtain learning data for model learning, and input data to which the learning model is to be applied. The input unit 140a may include a camera, a microphone, and / or a user input unit. The output unit 140b may generate output related to vision, hearing, or touch. The output unit 140b may include a display unit, a speaker, and / or a tactile module. The sensing unit 140 may use various sensors to obtain at least one of the internal information of the AI ​​device 100, the surrounding environment information of the AI ​​device 100, and the user information. The sensor unit 140 may include a proximity sensor, an illumination sensor, an acceleration sensor, a magnetic sensor, a gyroscope sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, a light sensor, a microphone, and / or a radar.

[0173] The learning processor unit 140c may use the learning data to learn a model composed of an artificial neural network. The learning processor unit 140c may perform AI processing together with the learning processor unit of the AI ​​server. The learning processor unit 140c may process information received from an external device through the communication unit 110 and / or information stored in the storage unit 130. In addition, the output value of the learning processor unit 140c may be sent to an external device through the communication unit 110c and may be stored in the storage unit 130.

[0174] The aforementioned embodiments are realized by combining the structural elements and features of the present disclosure in a predetermined manner. Unless specified separately, each structural element or feature should be selectively considered. Each structural element or feature can be performed without being combined with other structural elements or features. Moreover, some structural elements and / or features can be combined with each other to constitute an embodiment of the present invention. The order of operations described in the embodiments of the present invention can be changed. Some structural elements or features of an embodiment can be included in another embodiment, or can be replaced by the corresponding structural elements or features of another embodiment. Moreover, it will be apparent that some claims citing specific claims can be combined with other claims citing other claims other than specific claims to constitute an embodiment, or new claims can be added by way of amendment after submitting an application.

[0175] Those skilled in the art will appreciate that the present invention may be implemented in other specific ways different from the ways set forth herein without departing from the spirit and essential features of the present invention. Therefore, the above exemplary embodiments should be interpreted as exemplary and non-restrictive in all respects. The scope of the present invention should be determined by the appended claims and their legal equivalents, rather than by the above description, and all changes falling within the meaning and equivalent scope of the appended claims should be included therein.

Claims

1. A method for processing a multimedia signal by a device, the method comprising: obtaining at least one video object from the multimedia signal and obtaining at least one audio object from the multimedia signal, wherein the at least one video object represents a face; extracting video feature information for the at least one video object and audio feature information for the at least one audio object; Based on the video feature information and the audio feature information, determining, by an object matching engine, a correlation between the at least one video object and the at least one audio object; as well as The video feature information is extracted based on the following: a ratio between a distance between a top border of a first rectangular area including the face and a top border of a second rectangular area including lips within the first rectangular area, a height of the second rectangular area, and a distance between a bottom border of the second rectangular area and a bottom border of the first rectangular area; and a ratio between a distance between a left border of the second rectangular area and a left border of the first rectangular area, a width of the second rectangular area, and a distance between a right border of the second rectangular area and a right border of the first rectangular area, The first rectangular area is a face frame, and the second rectangular area includes a lip outline.

2. The method according to claim 1, wherein: Determining the correlation between the at least one video object and the at least one audio object includes: obtaining, by the object matching engine, information about a relationship between each of the at least one video object and a specific audio object of the at least one audio object based on the video feature information and the audio feature information; and Based on the information about the relationship between each of the at least one video object and the specific audio object, a specific video object related to the specific audio object is determined from among the at least one video object.

3. The method according to claim 2, wherein: Determining the specific video object related to the specific audio object includes: Based on the value of the information on the relationship being greater than a predetermined value, a video object related to the value of the information on the relationship is determined as the specific video object.

4. The method according to claim 3, wherein: Determining the specific video object related to the specific audio object further comprises: Based on the at least one video object including a plurality of video objects, a video object associated with a maximum value of the information on the relationship is determined from among the plurality of video objects as the specific video object.

5. The method according to claim 2, wherein: Determining a specific video object related to the specific audio object includes: Based on the value of the information on the relationship being smaller than a predetermined value, a video object related to the value of the information on the relationship is determined as the specific video object.

6. The method according to claim 5, wherein: Determining a specific video object related to the specific audio object further comprises: Based on the at least one video object including a plurality of video objects, a video object related to a minimum value of the information about the relationship is determined from among the plurality of video objects as the specific video object.

7. The method according to claim 2, wherein: The information about the relationship has a real value.

8. The method according to claim 1, wherein: The video feature information is extracted based on the vertical length and the horizontal length of the lip contour.

9. The method according to claim 1, wherein: The audio feature information is extracted based on linear prediction coding.

10. The method according to claim 1, wherein: The audio feature information is extracted based on a log-Mel filter bank.

11. The method according to claim 1, wherein: The audio feature information is extracted based on Mel-frequency cepstral coefficients.

12. The method according to claim 1, wherein: The audio feature information includes start information about the at least one audio object.

13. The method according to claim 1, wherein: The object matching engine includes a model trained based on learning.

14. An apparatus configured to process a multimedia signal, the apparatus comprising: a memory storing instructions; as well as At least one processor, the at least one processor is operably coupled to the memory and is configured to implement the following operations when executing the instructions, the operations comprising: obtaining at least one video object from the multimedia signal and obtaining at least one audio object from the multimedia signal, wherein the at least one video object represents a face; extracting video feature information for the at least one video object and audio feature information for the at least one audio object; Determining, by an object matching engine, a correlation between the at least one video object and the at least one audio object based on the video feature information and the audio feature information; and The video feature information is extracted based on the following: a ratio between a distance between a top border of a first rectangular area including a face and a top border of a second rectangular area including lips within the first rectangular area, a height of the second rectangular area, and a distance between a bottom border of the second rectangular area and a bottom border of the first rectangular area; and a ratio between a distance between a left border of the second rectangular area and a left border of the first rectangular area, a width of the second rectangular area, and a distance between a right border of the second rectangular area and a right border of the first rectangular area, The first rectangular area is a face frame, and the second rectangular area includes a lip outline.

Images