Method for matching voice according to objects included in video and computing device for performing method

Abstract

Disclosed is a method for matching voices according to objects included in a video, comprising the steps of: separating a plurality of voices included in the video; determining the diversity between the plurality of voices; on the basis of the diversity among the plurality of voices, selecting a part of duration in the whole duration of the video as a matching duration; matching the plurality of voices and the plurality of objects with each other on the basis of mouth movement of the plurality of objects included in the video within the matching duration; and matching the plurality of voices and the plurality of objects with each other in the whole duration of the video according to a matching result between the plurality of voices and the plurality of objects in the matching duration.

Description

Technical Field

[0001] The present disclosure relates to a method of matching sound sources respectively corresponding to objects in a video in which a plurality of objects appear and controlling reproduction of the video according to the matching result, and a computing device for executing the method. Background Art

[0002] As the environment for watching videos becomes more diverse, the number of methods for viewers to interact with videos is increasing. For example, when a video is reproduced through a screen that can perform touch input, such as a smart phone or a tablet personal computer (PC), the viewer can fix the focus of the video on a specific character by zooming in on a partial area of ​​the video (an area where a specific character appears) through touch input.

[0003] Thus, when the focus of the video is fixed on a specific character through user input, such focus may also be applied to the audio output corresponding to the specific character to provide intuitive feedback to the user. In this regard, it may be necessary to match the sound source (voice) of the video with the character appearing in the video. Summary of the invention

[0004] According to an embodiment of the present disclosure, a method for matching voice for each object included in a video includes: separating multiple voices included in the video, determining the dissimilarity between the multiple voices, selecting a partial duration of the entire duration of the video as a matching duration based on the dissimilarity between the multiple voices, matching the multiple voices with the multiple objects included in the video respectively within the matching duration based on the mouth movements of the multiple objects, and matching the multiple voices with the multiple objects respectively within the entire duration of the video based on the matching results between the multiple voices and the multiple objects within the matching duration.

[0005] According to an embodiment of the present disclosure, a computing device includes: an input / output interface configured to display a screen that reproduces a video and receive input from a user; a memory that stores a program for matching objects with voices; and a processor, wherein the processor is configured to execute the program to: separate multiple voices included in a video, determine the dissimilarity between the multiple voices, select a partial duration of the entire duration of the video as a matching duration based on the dissimilarity between the multiple voices, match the multiple voices with the multiple objects included in the video respectively within the matching duration based on the mouth movements of the multiple objects, and match the multiple voices with the multiple objects respectively within the entire duration based on the matching results between the multiple voices and the multiple objects within the matching duration.

[0006] According to an embodiment of the present disclosure, a computer-readable recording medium has recorded thereon a program for executing an embodiment of the method of the present disclosure on a computer.

[0007] According to an embodiment of the present disclosure, a computer program is stored in a computer-readable medium so as to execute an embodiment of the method of the present disclosure on a computer. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0009] Figure 1 is a diagram for describing a situation in which a sound source is matched for each object included in a video and reproduction of the video is controlled according to a matching result according to an embodiment of the present disclosure;

[0010] Figure 2 is a block diagram of a configuration of a computing device for matching a sound source for each object included in a video and controlling reproduction of the video according to a matching result according to an embodiment of the present disclosure;

[0011] Figure 3 is a diagram representing a program for executing a method of matching a sound source for each object included in a video in a plurality of modules according to an embodiment of the present disclosure;

[0012] Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 and Fig. 9 It is configuration Figure 3 A diagram of the detailed configuration of the program's modules.

[0013] Fig.10 , Fig.11 , Fig.12 , Fig.13 and Fig.14 is a flowchart of a method for matching a sound source for each object included in a video and controlling reproduction of the video according to a matching result according to an embodiment of the present disclosure;

[0014] Fig.15 is a diagram for describing in detail a method of selecting a portion of the entire duration of a video as a matching duration according to an embodiment of the present disclosure;

[0015] Fig.16 is a diagram for describing a method for extracting image features for a mouth movement of an object according to an embodiment of the present disclosure;

[0016] Fig.17is a table for describing an improvement in matching accuracy when matching between an object and a sound source is performed within a selected matching duration according to an embodiment of the present disclosure; and

[0017] Fig.18 , Fig.19 and Fig. 20 is a diagram for describing an embodiment of the present disclosure for controlling reproduction of a video based on a result of matching a sound source for each object included in the video according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] When describing the present disclosure, descriptions of technical contents that are well known in the technical field to which the present disclosure belongs and are not directly related to the present disclosure will be omitted. By omitting unnecessary descriptions, the gist of the present disclosure can be conveyed more clearly without blurring the subject matter. The terms used below are defined in consideration of the functions in the present disclosure and may have different meanings according to the intentions, habits, etc. of the user or operator. Therefore, the terms used here must be defined based on the meaning of the terms and the descriptions throughout the specification.

[0019] For the same reason, for the sake of clarity, each component may be exaggerated, omitted or schematically shown in the drawings. Moreover, the size of each component does not fully reflect the actual size. In the drawings, the same reference numerals represent the same elements.

[0020] By referring to the following detailed description and drawings of the embodiments of the present disclosure, the advantages and features of the present disclosure and the methods for realizing the same can be more easily understood. However, the present disclosure is not limited to the embodiments of the present disclosure described below, and can be implemented in various forms. The embodiments of the present disclosure are provided to fully convey the present disclosure and fully inform the scope of the present disclosure to those of ordinary skill in the art. The embodiments of the present disclosure can be defined according to the claims. Throughout the specification, the same reference numerals represent the same elements. Although the embodiments of the present disclosure are described, when it is determined that the detailed description of the relevant functions or configurations may unnecessarily obscure the subject matter of the present disclosure, the detailed description thereof is omitted. The terms used below are defined in consideration of the functions in the present disclosure, and may have different meanings according to the intentions, habits, etc. of the user or operator. Therefore, the terms used here must be defined together based on the meaning of the terms and the descriptions throughout the specification.

[0021] According to an embodiment of the present disclosure, the combination of blocks in a flowchart or process flow chart can be executed by computer program instructions. These computer program instructions can be loaded into a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, and the instructions executed by the processor of the computer or other programmable data processing device can generate means for implementing the functions specified in the flowchart frame. The computer program instructions can be stored in a computer executable or computer readable memory that can instruct a computer or another programmable data processing device to implement functions in a specific manner, and the instructions stored in the computer executable or computer readable memory can also generate a manufacturing item containing an instruction unit for executing the functions described in the flowchart frame. The computer program instructions can be loaded into a computer or another programmable data processing device.

[0022] In addition, each frame of the flow chart may represent a module, segment or part of code including one or more executable instructions for performing a specified logical function. According to an embodiment of the present disclosure, the functions in the frame may not be performed in sequence. For example, depending on the functions involved, two frames shown in succession may actually be performed substantially simultaneously or in reverse order.

[0023] Throughout the disclosure, the expression "at least one of a, b, or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0024] The term "unit" or "~device / operator" in the embodiments of the present disclosure may indicate a software component or a hardware component such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), and performs a specific function. The term "unit" or "~device / operator" is not limited to software or hardware. The "unit" or "~device / operator" may be formed in an addressable storage medium, or may be formed to operate one or more processors. According to an embodiment of the present disclosure, the term "unit" or "~device / operator" may refer to components such as software components, object-oriented software components, class components, and task components, and may include processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcodes, circuits, data, databases, data structures, tables, arrays, or variables. The functions provided by a specific component or a specific "unit" or "~device / operator" may be combined to reduce their number or divided into additional components. In addition, in an embodiment of the present disclosure, a "unit" or "~device / operator" may include at least one processor.

[0025] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0026] Figure 1is a diagram for describing a case in which a sound source is matched for each object included in a video and reproduction of the video is controlled according to a matching result according to an embodiment of the present disclosure.

[0027] Figure 1 The first screen 100a corresponds to a scene of the video being reproduced, and the second screen 100b is an enlarged image of a partial area 10 of the first screen 100a. There are two characters 1 and 2 in the first screen 100a, and hereinafter, the characters appearing in the video will be referred to as "objects" (included) in the video.

[0028] When both the first object 1 and the second object 2 speak in the video, the voices of the first object 1 and the second object 2 will be mixed and output. Hereinafter, voice and sound source are used with the same meaning.

[0029] like Figure 1 As shown, when the partial area 10 is enlarged from the first screen 100a to display the second screen 100b, the present disclosure emphasizes and outputs the voice of the first object 1. In this regard, the sound source and the object included in the video need to match each other. Therefore, the method of matching the corresponding sound source for each object included in the video will be described in detail below, and then an embodiment of the present disclosure for controlling the reproduction of the video based on the matching result will be described.

[0030] Figure 2 is a block diagram of a configuration of a computing device for matching a sound source for each object included in a video and controlling reproduction of the video according to a matching result according to an embodiment of the present disclosure.

[0031] exist Figure 2 In the embodiment, the computing device 200 shown may be a display device (e.g., a smart phone or a tablet personal computer (PC)) that reproduces a video, or a separate server connected to the display device through wired or wireless communication. The method for matching a sound source for each of a plurality of objects included in a video according to an embodiment of the present disclosure may be performed by the display device that reproduces a video, may be performed by a separate server connected to the display device, or may be performed jointly by the display device and the server (the two devices divide and perform the processing included in the method).

[0032] In the following, for the convenience of description, it is assumed that Figure 2The computing device 200 is a display device that reproduces a video, and performs a method for matching a sound source for each of a plurality of objects included in a video according to an embodiment of the present disclosure. However, as described above, the embodiments of the present disclosure are not limited thereto, and it is obvious that there is a separate server connected to the display device, and the server can perform some or all of the processing. Therefore, it should be explained that among the operations performed by the computing device 200 according to an embodiment of the present disclosure described below, operations other than the operation of displaying an image on a screen can also be performed by a separate computing device (such as a server), even if there is no special description.

[0033] Reference Figure 2 According to an embodiment of the present disclosure, the computing device 200 may include a communication interface 210, an input / output interface 220, a processor 230, and a memory 240. However, the components of the computing device 200 are not limited thereto, and the computing device 200 may include more or fewer components than the above components. According to an embodiment of the present disclosure, some or all of the communication interface 210, the input / output interface 220, the processor 230, and the memory 240 may be implemented in the form of one chip, and the processor 230 may include one or more processors.

[0034] The communication interface 210 is a component for transmitting / receiving signals (control commands and data) to / from an external device via wire or wirelessly, and may include a communication chipset supporting various communication protocols. The communication interface 210 may receive an external signal and output it to the processor 230, or may transmit a signal output from the processor 230 to the outside.

[0035] The input / output interface 220 may include an input interface (e.g., a touch screen, a hard button, or a microphone) for receiving a control command or information from a user and an output interface (e.g., a display panel or a speaker) for displaying a state of the computing device 200 or a result of performing an operation according to the control of the user. According to an embodiment of the present disclosure, the input / output interface 220 may display a video being reproduced, and receive an input of enlarging a partial area of ​​the video or an input of selecting a specific object included in the video from the user.

[0036] The processor 230 is configured to control a series of operations so that the computing device 200 can operate according to the embodiments of the present disclosure described below, and the processor 230 may include one or more processors. In this case, the one or more processors may be a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), or a digital signal processor (DSP)), a dedicated graphics processor (such as a graphics processing unit (GPU) or a visual processing unit (VPU)), or a dedicated AI processor (such as a neural processing unit (NPU)). For example, when one or more processors are dedicated artificial intelligence (AI) processors, the dedicated AI processor may be designed with a hardware structure dedicated to processing a specific AI model.

[0037] The processor 230 may record data in the memory 240 or read data stored in the memory 240, and in particular, may process data according to predefined operation rules or AI models by executing a program stored in the memory 240. Therefore, the processor 230 may perform the operations described in the embodiments of the present disclosure below, and unless otherwise described, the operations described in the embodiments of the present disclosure below as being performed by the computing device 200 may be considered to be performed by the processor 230.

[0038] The memory 240 is configured to store various programs or data, and may include a storage medium such as a read-only memory (ROM), a random access memory (RAM), a hard disk, a CD-ROM or a DVD, or a combination of storage media. The memory 240 may not be provided separately, and may be included in the processor 230. The memory 240 may be configured as a volatile memory, a non-volatile memory, or a combination of a volatile memory and a non-volatile memory. The memory 240 may store a program for performing operations according to the embodiments of the present disclosure described below. The memory 240 may provide the stored data to the processor 230 according to the request of the processor 230.

[0039] Hereinafter, an embodiment of the present disclosure will be described in detail, in which the computing device 200 matches a sound source for each of a plurality of objects included in a video and controls reproduction of the video according to a result of the matching.

[0040] Figure 3 is a diagram representing a program for executing a method of matching a sound source for each object included in a video in a plurality of modules according to an embodiment of the present disclosure, and Figures 4 to 9 It is configuration Figure 3 The detailed configuration diagram of the modules of the program 300 can be obtained by classifying the operations performed when the processor 230 executes the program 300 stored in the memory 240 according to the function. Figure 3Therefore, in the following, the description is made of modules 310 to 370. Figure 3 The operations performed by modules 310 to 370 may actually be performed by processor 230 .

[0041] Reference Figure 3 According to an embodiment of the present disclosure, a program 300 for matching a sound source for each of a plurality of objects included in a video may include: a sound source separation module 310 based on audio information, a matching duration selection module 320, an audio feature extraction module 330, a ROI extraction module 340, a mouth movement feature extraction module 350, a heterogeneous information feature matching module 360, and an object-sound source matching module 370.

[0042] Reference Figures 10 to 14 The flowchart is described by Figures 3 to 9 The operations performed by the modules 310 to 370 shown in FIG. In the embodiments of the present disclosure described below, a video includes M sound sources (voices) and N objects.

[0043] Figures 10 to 14 is a flowchart of a method of matching a sound source for each object included in a video and controlling reproduction of the video according to a matching result according to an embodiment of the present disclosure.

[0044] refer to Fig.10 In operation 1001, the processor 230 of the computing device 200 may separate a plurality of sound sources included in the video. Operation 1001 may be performed by Figure 3 The sound source separation module 310 based on audio information is performed. The sound source separation module 310 based on audio information can separate a plurality of sound sources by using only audio information included in the video.

[0045] Figure 4 The detailed configuration of the sound source separation module 310 based on audio information is shown in FIG. Figure 4 According to an embodiment of the present disclosure, the sound source separation module 310 based on audio information may include an encoder ENC, a mask generation and a decoder DEC. Figure 3 and Figure 4 As shown in , the audio information-based sound source separation module 310 can receive an audio input (audio information included in a video) and separate a plurality of sound sources Voice#1, ..., Voice#m, ..., Voice#M therefrom.

[0046] 1. Choice of match duration

[0047] In operation 1002, the processor 230 may determine the dissimilarity between the plurality of sound sources, and in operation 1003, the processor 230 may select a partial duration in the entire duration of the video as a matching duration based on the result of determining the dissimilarity between the plurality of sound sources. Here, the "matching duration" refers to the duration of performing the process of matching the object with the sound source, in particular, the duration of comparing the features (image features and audio features) extracted from the object and the sound source to match the object with the sound source. In other words, according to an embodiment of the present disclosure, the processor 230 performs the operation (feature comparison) for matching the object with the sound source only within the selected matching duration rather than within the entire duration of the video. By performing the matching operation only within the matching duration rather than within the entire duration of the video, the throughput can be effectively reduced, and by setting the matching duration as described below, the high matching accuracy can be maintained.

[0048] Operation 1002 and operation 1003 can be performed by Figure 3 The matching duration selection module 320 is executed. In order to improve the matching accuracy while matching the object contained in the video with the sound source, the difference between the object and the sound source needs to be high (for example, when only one character speaks and the other character does not speak, or when the speaking timing between the characters is misaligned). Therefore, according to an embodiment of the present disclosure, in order to improve the matching accuracy between the object and the sound source, the matching duration selection module 320 can select the duration with the highest difference between the multiple sound sources as the matching duration.

[0049] The detailed configuration of the matching duration selection module 320 is described in Figure 5 See Figure 5 According to an embodiment of the present disclosure, the matching duration selection module 320 may include a primary standard deviation (STD) calculation module 321, a secondary STD calculation module 322, and a selection module 323. The matching duration selection module 320 may receive a plurality of sound sources Voice#1, ..., Voice#m, ..., Voice#M separated by the sound source separation module 310 based on audio information, select a matching duration τ therefrom, and output the matching duration τ. The process of selecting the matching duration by the matching duration selection module 320 will refer to Fig.11 , Fig.12 and Fig.15 Describe in detail.

[0050] Fig.11 is included in Fig.10 Flowchart of detailed operations in operation 1002 and operation 1003. Fig.11In operation 1101, the matching duration selection module 320 divides the entire duration of the video into a plurality of durations. Here, the length of the duration can be set differently as needed.

[0051] In operation 1102, the matching duration selection module 320 calculates a dissimilarity value between a plurality of sound sources for each of a plurality of durations. Here, the “dissimilarity value” is a numerical value indicating the degree of dissimilarity of a sound source, and indicates that the higher the dissimilarity value, the higher the degree of dissimilarity of the sound source, that is, there is a large difference between the sound sources. Various methods may be used to calculate the dissimilarity value as an index indicating the degree of dissimilarity, and according to an embodiment of the present disclosure, the matching duration selection module 320 may calculate the STD of the sound source for each duration, calculate the STD among the STDs of different sound sources, and use the STD as the dissimilarity value.

[0052] Figure 5 The primary STD calculation module 321 and the secondary STD calculation module 322 of the match duration selection module 320 may calculate the dissimilarity values ​​as described above. Fig.12 Shown in Fig.11 The detailed operations included in operation 1102 are shown in FIG. Fig.12 In operation 1201, the primary STD calculation module 321 may calculate the STD of the plurality of sound sources for each of the plurality of durations. In operation 1202, the secondary STD calculation module 322 may calculate the STD of the plurality of sound sources as a dissimilarity value for each of the plurality of durations. A method of calculating the STD of the STD of the sound source as a dissimilarity value will now be described with reference to a specific embodiment of the present disclosure.

[0053] Fig.15 is a diagram for describing in detail a method of selecting a portion of the entire duration of a video as a matching duration according to an embodiment of the present disclosure, and in detail, is a diagram showing an example of calculating STD of a sound source as a dissimilarity value.

[0054] refer to Fig.15 , a first graph 1510 is a graph showing an output of a first sound source Voice#1, and a second graph 1520 is a graph showing an output of a second sound source Voice#2. A process of calculating a difference value between the first sound source Voice#1 and the second sound source Voice#2 is as follows.

[0055] First, the matching duration selection module 320 calculates the STD of the first sound source Voice#1 and the second sound source Voice#2 for each duration. This corresponds to the above Figure 5 The main STD calculation module 321 performs Fig.12 Operation 1201 of . The result of calculating the STD of the first sound source Voice# 1 and the second sound source Voice# 2 is shown as “STD (Voice# 1 )” and “STD (Voice# 2 )” in the third graph 1530 .

[0056] Then, the matching duration selection module 320 calculates the STD of the first sound source Voice#1 and the STD of the second sound source Voice#2 for each duration. This corresponds to the above Figure 5 The secondary STD calculation module 322 performs Fig.12 Operation 1202 of FIG. 15 is performed to calculate the STD of the first sound source Voice# 1 and the STD of the second sound source Voice# 2 . The result of calculating the STD of the first sound source Voice# 1 and the STD of the second sound source Voice# 2 is shown as “STD of STD” in the third graph 1530 .

[0057] Return to reference Fig.11 In operation 1103, the selection module 323 of the matching duration selection module 320 may select and output the duration with the highest dissimilarity value as the matching duration τ by comparing the dissimilarity values ​​calculated for each duration. Fig.15 In the third graph 1530, the duration with the highest dissimilarity value is indicated as the first point 1531, and the duration with the lowest dissimilarity value is indicated as the second point 1532. When the result of calculating the dissimilarity value is the same as the third graph 1530, the selection module 323 may select the first point 1531 as the matching duration τ.

[0058] Experiments have been performed to verify the effect of enhancing the matching accuracy by selecting the duration with the highest dissimilarity value as the matching duration τ, and the results are shown in Fig.17 In. Fig.17 In Table 1700 of FIG. 1700 , “minimum error duration” corresponds to the duration with the lowest dissimilarity value, i.e., Fig.15 In addition, the "maximum error duration" in table 1700 corresponds to the duration with the highest dissimilarity value, that is, Fig.15 The first point 1531 in the third figure 1530.

[0059] exist Fig.17 In Table 1700, the numerical values ​​shown in Table 1700 correspond to the probabilities that the first object Object#1 and the second object Object#2 match the first sound source Voice#1 and the second sound source Voice#2, respectively, and the matching probability calculated by feature comparison in the minimum error duration and the matching probability calculated by feature comparison in the maximum error duration are both indicated.

[0060] Referring to the first object Object#1, as a result of calculating the matching probability by feature comparison in the minimum error duration, it is determined that the probability that the first sound source Voice#1 can correspond to the first object Object#1 is 0.6134, and the probability that the second sound source Voice#2 can correspond to the first object Object#1 is 0.3141. As a result of calculating the matching probability by feature comparison in the maximum error duration for the same first object Object#1, it is determined that the probability that the first sound source Voice#1 can correspond to the first object Object#1 is 0.9861, and the probability that the second sound source Voice#2 can correspond to the first object Object#1 is 0.0051.

[0061] Comparing the experimental results in the two durations, it is determined that the matching probability calculated by feature comparison in the maximum error duration further accurately indicates the matching sound source. The probability that the first sound source Voice#1 can match the first object Object#1 is also higher in the minimum error duration, but the difference between the probabilities is greater in the maximum error duration. In the experimental results on the second object Object#2, it is also determined that the matching probability calculated by feature comparison in the maximum error duration is further clearly distinguished.

[0062] The matching duration selection module 320 can output the selected matching duration τ to the mouth movement feature extraction module 350, and can receive multiple sound sources Voice#1,..., Voice#m,..., Voice#M, and output multiple sound sources Voice#1,τ,..., Voice#m,τ,..., Voice#M,τ regarding the matching duration τ.

[0063] 2. Sound source matching based on the subject’s mouth movement during the matching time

[0064] Return to reference Fig.10 In operation 1004, the processor 230 may match multiple objects included in multiple sound sources and videos with each other based on the mouth movements of the multiple objects within the matching duration. Here, "mouth movement" is a broader concept than the shape or movement of the speaker's lips, and includes not only changes in the lip shape of the object (speaker), but also changes in the facial shape around the mouth of the object. Matching is performed based on the mouth movement of the broad concept defined in this way to prevent the problem of reduced matching accuracy when occlusion occurs, in which a portion of the mouth of the object is covered (for example, when the speaker speaks close to the microphone). A specific method for performing matching based on mouth movement will be described in detail below.

[0065] Operation 1004 may be performed by Figure 3 The audio feature extraction module 330, the ROI extraction module 340, the mouth movement feature extraction module 350, the heterogeneous information feature matching module 360 ​​and the object-sound source matching module 370 are used to perform. Fig.13 , 14 and 16 describe operation 1004 in detail.

[0066] Fig.13 is included in Fig.10 1004. Fig.13 In operation 1301, the processor 230 may extract image features of the mouth movements of multiple objects within the matching duration. Operation 1301 may be performed by Figure 3 The ROI extraction module 340 and the mouth movement feature extraction module 350 are used for execution.

[0067] Fig.14 is included in Fig.13 Operation 1401 may be performed by the ROI extraction module 340 , and operation 1402 may be performed by the mouth movement feature extraction module 350 .

[0068] In operation 1401, the ROI extraction module 340 may extract a facial region including a plurality of objects around the mouth as an ROI image. Figure 7 As shown in . The ROI extraction module 340 can receive a video input (video information included in the video) and extract a plurality of ROI images ROI#1, ..., ROI#n, ..., ROI#N therefrom. As described above, in the present embodiment of the present disclosure, the mouth movement feature extraction module 350 described below even extracts the feature of the change 1 of the facial shape around the mouth of the object in order to prevent the problem of reduced matching accuracy when occlusion occurs, so the ROI extraction module 340 can extract the ROI to include the entire mouth of the object, and also include the area around the mouth within a certain distance from the mouth.

[0069] In operation 1402, the mouth movement feature extraction module 350 may extract image features only from the remaining area of ​​the ROI image excluding at least a portion of the area of ​​the ROI image, and may repeatedly extract image features while setting at least a portion of the area excluded from the ROI image differently. The detailed configuration of the mouth movement feature extraction module 350 is described in Figure 8 The mouth movement feature extraction module 350 can extract image features F from multiple ROI images ROI#1, ..., ROI#n, ..., ROI#N during the matching duration τ. ROI#1,τ ,……,F ROI#n,τ ,……,FROI#N,τ In detail, the mouth movement feature extraction module 350 receives a plurality of ROI images ROI#1, ..., ROI#n, ..., ROI#N extracted from the entire duration of the video from the ROI extraction module 340, but may receive information about the matching duration τ from the matching duration selection module 320, and extracts image features of only the ROI images located within the matching duration τ.

[0070] The mouth movement feature extraction module 350 can extract features of the plurality of ROI images ROI#1, ..., ROI#n, ..., ROI#N not only from the entire region but also from the remaining region excluding the partial region. Fig.14 The flowchart only includes operation 1402, in which image features are extracted only from the remaining area of ​​the ROI excluding at least a portion of the area of ​​the ROI (there is no operation of extracting image features from the entire area of ​​the ROI), but it does not exclude that the mouth movement feature extraction module 350 extracts image features from the entire area of ​​the ROI, but it can be selectively included.

[0071] The mouth movement feature extraction module 350 may extract image features after performing region-ware feature filtering on the provided ROI image, and may repeatedly extract image features from the same ROI image while variously changing the region used for the region-ware feature filtering. Fig.16 A method of extracting mouth movement features from ROI images ROI#1, ..., ROI#n, ..., ROI#N is described in detail.

[0072] Fig.16 Detailed description of a method for extracting image features for a subject's mouth movement according to an embodiment of the present disclosure. Fig.16 , the ROI extraction module 340 may extract a region including the entire mouth of the subject and including a region around the mouth within a certain distance from the mouth as a ROI image 1610 from the entire image 1600 .

[0073] exist Fig.16 The right side of FIG. 1 shows ROI images 1610a, 1610b, and 1610c, in which the area to be filtered is slightly changed. The gray area in the ROI images 1610a, 1610b, and 1610c indicates the area excluded during image feature extraction (hereinafter, referred to as the filtering area). Fig.16The filter area shown in is only an example, and the filter area can be set in various ways. The mouth movement feature extraction module 350 can extract image features from the entire ROI image 1610, but can also extract image features from the ROI images 1610a, 1610b, and 1610c in which partial areas are filtered, and combine the extracted image features to output the image feature F ROI#1,τ ,……,F ROI#n,τ ,……,F ROI#N,τ .

[0074] In this way, the mouth movement feature extraction module 350 can extract image features while slightly changing the target area in the ROI image 1610 and combine the results, thereby extracting image features F that reflect not only the changes in the lip shape of the subject but also the changes in the facial shape around the mouth. ROI#1,τ ,……,F ROI#m,τ ,……,F ROI#N,τ Therefore, even when occlusion occurs due to a portion of the subject's mouth being covered by another object, high matching accuracy can be maintained.

[0075] Return to reference Fig.13 In operation 1302, the processor 230 may extract audio features from multiple sound sources within the matching duration. Operation 1302 may be performed by the audio feature extraction module 330, and the detailed configuration of the audio feature extraction module 330 is described in Figure 6 The audio feature extraction module 330 can extract audio features F from multiple sound sources Voice#1,τ, ..., Voice#m,τ, ..., Voice#M,τ of matching duration τ. Voice#1,τ ,……,F Voice#m,τ ,……,F Voice#M,τ .

[0076] In operation 1303, the processor 230 may process the image features F extracted in the previous operation. ROI#1,τ ,……,F ROI#m,τ ,……,F ROI#N,τ and audio feature F Voice#1,τ ,……,F Voice#m,τ ,……,F Voice#M,τ Operation 1303 may be performed by the heterogeneous information feature matching module 360, and the detailed configuration of the heterogeneous information feature matching module 360 ​​is as follows: Fig. 9 Shown in.

[0077] refer to Fig. 9 The heterogeneous information feature matching module 360 ​​can be used to extract image features F from an ROI image of an object. ROI#n,τ and the audio features F extracted from a sound source voice#m,τPerform feature matching to output the probability P (F voice#m,τ | F ROI#n,τ ).

[0078] When matching is performed on different types of information (image features and audio features), the matching accuracy may be reduced due to the feature differences between the information, and the heterogeneous information feature matching module 360 ​​can use the time adaptive vector to prevent the reduction of the matching accuracy. The function and technical meaning of the time adaptive vector will now be described in detail.

[0079] Video and audio usually have very different resolutions on the time axis. For example, the general video resolution is 30 fps, while the general audio resolution is 48 kHz, which is significantly higher than the video resolution. Due to such a large resolution difference, the audio often changes while the video hardly changes (for example, another voice is continuously output while the lip shape hardly changes). Therefore, when matching video features (image features) and audio features over a duration where the video changes rapidly, the matching accuracy increases.

[0080] Therefore, according to an embodiment of the present disclosure, the heterogeneous information feature matching module 360 ​​can generate a vector (time adaptive vector) for providing greater weight to the duration of a video that changes rapidly based on video features (image features), and improve the matching accuracy by using the generated vector to assign weights to features of a specific duration (video features and audio features).

[0081] The heterogeneous information feature matching module 360 ​​can match the image feature F by measuring the similarity between two features. ROI#n,τ and audio feature F Voice#m,τ Perform feature matching. For example, the heterogeneous information feature matching module 360 ​​can calculate the distance between two features by any of various methods (e.g., Euclidean distance, Manhattan distance, or Minkowski distance), and determine that the similarity is high when the distance is short. ROI#n,τ With audio feature F Voice#m,τ When the similarity between them is high, the matching probability between the two features can be calculated as high.

[0082] In summary, the heterogeneous information feature matching module 360 ​​can measure the image feature F ROI#n,τ With audio feature F Voice#m,τ The similarity between them is greater, while assigning greater weight to the image feature F ROI#n,τThe similarity within the duration having a greater variation over time is calculated, and a probability that each of the plurality of sound sources can correspond to each of the plurality of objects is calculated based on the measured similarity.

[0083] In other words, the heterogeneous information feature matching module 360 ​​can allocate a larger weight to the image feature F ROI#n,τ The change of image features F within a larger duration ROI#n,τ and audio feature F Voice#m,τ to perform matching, and calculate a probability that each of the plurality of sound sources may correspond to each of the plurality of objects based on a result of performing the matching.

[0084] Return to reference Fig.13 In operation 1304, the processor 230 may match a sound source corresponding to each of the plurality of objects from the plurality of sound sources based on the result of the feature matching. Operation 1304 may be performed by the object-sound source matching module 370, and the object-sound source matching module 370 may select an object-sound source pair with the highest matching probability according to the following equation 1.

[0085] [Equation 1]

[0086]

[0087] The computing device 200 may match a corresponding sound source for each object included in the video by performing the above process.

[0088] Return to reference Fig.10 In operation 1005, the processor 230 may match the multiple objects with the multiple sound sources within the entire duration of the video based on the matching results between the multiple sound sources and the multiple objects in the matching duration. In other words, the processor 230 performs matching by comparison between the image features and the audio features only in the matching duration which is a partial duration of the entire duration of the video, and extends and applies the matching results to the entire duration, thereby enabling matching between the sound sources and the objects at a relatively low throughput.

[0089] 3. Control video playback based on matching results

[0090] In operation 1006 , the processor 230 may control reproduction of the video based on the matching result so that a plurality of sound sources are output in association with a plurality of objects. Fig.10 Operation 1006 is indicated with a dashed line to indicate that operation 1006 is optional.

[0091] Figures 18 to 20 is a diagram for describing an embodiment of the present disclosure for controlling reproduction of a video based on a result of matching a sound source for each object included in the video.

[0092] According to an embodiment of the present disclosure, the computing device 200 may control the sound sources matching the multiple objects to be allocated to the multiple speakers based on the positions of the multiple objects in the video, such as Fig.18 shown.

[0093] refer to Fig.18 , the first object 1 and the second object 2 included in the reproduction screen 1800 output voices respectively, and the first voice Voice#1 corresponds to the first object 1, and the second voice Voice#2 corresponds to the second object 2.

[0094] The computing device 200 may execute Fig.10 Operations 1001 to 1003 are performed to output the result of matching the first voice Voice#1 with the first object 1 and matching the second voice Voice#2 with the second object 2.

[0095] In the reproduction screen 1800, the first object 1 is located on the right side, and the second object 2 is located on the left side, so the computing device 200 can amplify and output the second voice Voice#2 through the left speaker 1810L, and amplify and output the first voice Voice#1 through the right speaker 1810R. Therefore, the viewer can better feel the realism or stereoscopic effect of the sound by outputting the voice according to the position of the object.

[0096] According to the embodiments of the present disclosure, Fig.19 As shown, when an input selecting one of a plurality of objects is received while reproducing a video, the computing device 200 may control the output of a sound source corresponding to the selected object among a plurality of sound sources to emphasize.

[0097] refer to Fig.19 , the first object 1 and the second object 2 included in the reproduction screen 1900 output voices respectively, and the first voice Voice#1 corresponds to the first object 1, and the second voice Voice#2 corresponds to the second object 2.

[0098] The computing device 200 may execute Fig.10 Operations 1001 to 1004 are performed to output the result of matching the first voice Voice#1 with the first object 1 and matching the second voice Voice#2 with the second object 2.

[0099] When the viewer selects the first object 1 through the selection unit 1920 (such as a finger or a mouse cursor), the computing device 200 can control the first voice Voice#1 matched with the first object 1 to be emphasized and output. Therefore, the first voice Voice#1 is amplified and output from both the left speaker 1910L and the right speaker 1910R.

[0100] exist Fig.19 , both the first voice Voice#1 and the second voice Voice#2 are output from the left speaker 1910L and the right speaker 1910R, and only the first voice Voice#1 is amplified and output, but on the other hand, when the first object 1 is selected, the computing device 200 can control the output of only the first voice Voice#1, excluding the second voice Voice#2.

[0101] According to the embodiments of the present disclosure, Fig. 20 As shown, when an input of zooming in a region displaying one of a plurality of objects is received while reproducing a video, the computing device 200 may control output of a sound source corresponding to the object included in the zoomed-in region among a plurality of sound sources for emphasis.

[0102] Fig. 20 The reproduction screen 2000 is enlarged Fig.19 A screen obtained by zooming in on an area of ​​the reproduction screen 1900 including the first object 1. Assume that the first object 1 and the second object 2 included in the entire screen before zooming in respectively output voices, and the first voice Voice#1 corresponds to the first object 1, and the second voice Voice#2 corresponds to the second object 2.

[0103] The computing device 200 may execute Fig.10 Operations 1001 to 1003 are performed to output the result of matching the first voice Voice#1 with the first object 1 and matching the second voice Voice#2 with the second object 2.

[0104] When the reproduction screen 2000 is displayed while the viewer zooms in on the area including the first object 1 through a touch input or the like, the computing device 200 may control the first voice Voice#1 matched with the first object 1 to be emphasized and output. Therefore, the first voice Voice#1 is output only from both the left speaker 2010L and the right speaker 2010R.

[0105] exist Fig. 20 In the example, in addition to the second voice Voice#2, the first voice Voice#1 is output only from the left speaker 2010L and the right speaker 2010R, but on the other hand, the computing device 200 can output both the first voice Voice#1 and the second voice Voice#2 from the left speaker 2010L and the right speaker 2010R while amplifying only the first voice Voice#1.

[0106] According to an embodiment of the present disclosure, a method for matching voice for each object included in a video includes: determining the dissimilarity between multiple voices, selecting a partial duration of the entire duration of the video as a matching duration based on the dissimilarity between the multiple voices, matching the multiple voices with the multiple objects included in the video respectively within the matching duration based on the mouth movements of the multiple objects, and matching the multiple voices with the multiple objects respectively within the entire duration of the video based on the results of the matching between the multiple voices and the multiple objects within the matching duration.

[0107] According to an embodiment of the present disclosure, the mouth movement may include a change in the lip shape of the plurality of subjects or a change in the facial shape around the mouth of the plurality of subjects.

[0108] According to an embodiment of the present disclosure, selecting a partial duration of the entire duration of a video as a matching duration may include: dividing the entire duration of the video into multiple durations, calculating a dissimilarity value between multiple voices for each of the multiple durations, and selecting a duration with the highest calculated dissimilarity value as the matching duration.

[0109] According to an embodiment of the present disclosure, calculating the dissimilarity value may include: calculating a standard deviation of the plurality of voices for each of the plurality of durations, and calculating a standard deviation of the standard deviations of the plurality of voices for each of the plurality of durations as the dissimilarity value.

[0110] According to an embodiment of the present disclosure, matching multiple voices with multiple objects included in a video respectively within a matching duration may include: extracting image features of mouth movements of the multiple objects from the matching duration, extracting audio features of the multiple voices from the matching duration, performing feature matching on the image features and the audio features, and matching the voices corresponding to the multiple objects respectively from among the multiple voices based on the results of the feature matching.

[0111] According to an embodiment of the present disclosure, performing feature matching includes: measuring the similarity between image features and audio features, wherein a greater weight is assigned to the similarity over a duration in which the image features change more over time, and based on the measured similarity, calculating the probability that multiple voices correspond to multiple objects respectively.

[0112] According to an embodiment of the present disclosure, performing feature matching includes: measuring at least a first similarity and a second similarity between an image feature and an audio feature, wherein a first weight is assigned to the first similarity and a second weight is assigned to the second similarity, wherein a first change in the image feature of the first similarity within a first duration is greater than a second change in the image feature of the second similarity within a second duration, the first weight is greater than the second weight, and based on the measured similarities, determining the probability that multiple voices correspond to multiple objects respectively.

[0113] According to an embodiment of the present disclosure, performing feature matching may include performing matching by assigning greater weights to image features and audio features in a section where changes in image features are greater, and calculating probabilities that multiple voices respectively correspond to multiple objects based on a result of performing matching.

[0114] According to an embodiment of the present disclosure, performing feature matching may include: performing matching by assigning a first weight to a first image feature and a first audio feature, and assigning a second weight to a second image feature and a second audio feature, wherein a first change in the first image feature in the first part is greater than a second change in the second image feature, and the first weight is greater than the second weight, and based on the result of performing the matching, determining the probability that multiple voices correspond to multiple objects respectively.

[0115] According to an embodiment of the present disclosure, extracting image features of mouth movement may include: extracting a facial area around the mouths of multiple objects as a region of interest (ROI), and extracting image features only from a remaining area of ​​the ROI excluding at least a portion of the ROI.

[0116] According to an embodiment of the present disclosure, extracting the image features only from the remaining region may include: repeatedly extracting the image features while differently setting at least a portion of the region excluded from the ROI.

[0117] According to an embodiment of the present disclosure, the method may further include: based on the result of the matching in the entire duration, controlling the reproduction of the video so that a plurality of voices are output in association with a plurality of objects.

[0118] According to an embodiment of the present disclosure, controlling reproduction of a video may include emphasizing output of a first voice corresponding to the first object among a plurality of voices when an input of selecting a first object from a plurality of objects is received during reproduction of the video.

[0119] According to an embodiment of the present disclosure, an input of selecting a first object from among a plurality of objects may be in the form of zooming in or touching an area where the first object is displayed on a screen where a video is being reproduced.

[0120] According to an embodiment of the present disclosure, controlling the reproduction of a video may include controlling voices matching the plurality of objects to be distributed to a plurality of speakers based on positions of the plurality of objects in the video.

[0121] According to an embodiment of the present disclosure, a computing device for matching voice for each object included in a video includes: an input / output interface 220, configured to display a screen that reproduces a video and receive input from a user, a memory 240, storing a program for matching objects and voices, and a processor 230, wherein the processor 230 is configured to: execute the program to separate multiple voices included in the video, determine the dissimilarity between the multiple voices, select a partial duration of the entire duration of the video as a matching duration based on the dissimilarity between the multiple voices, match the multiple voices with the multiple objects respectively within the matching duration based on the mouth movements of the multiple objects included in the video, and match the multiple voices with the multiple objects respectively within the entire duration of the video based on the matching results between the multiple voices and the multiple objects within the matching duration.

[0122] According to an embodiment of the present disclosure, the mouth movement may include: a change in lip shapes of the plurality of objects or a change in facial shapes around the mouths of the plurality of objects.

[0123] According to an embodiment of the present disclosure, while selecting a partial duration of the entire duration of the video as the matching duration, the processor 230 can also be configured to: divide the entire duration of the video into multiple durations, calculate the dissimilarity value between multiple voices for each of the multiple durations, and select the duration with the highest calculated dissimilarity value as the matching duration.

[0124] According to an embodiment of the present disclosure, when calculating the dissimilarity value, the processor 230 may also be configured to: calculate (determine) the standard deviation of the plurality of voices for each of the plurality of durations, and calculate (determine) the standard deviation of the standard deviations of the plurality of voices for each of the plurality of durations as the dissimilarity value.

[0125] According to an embodiment of the present disclosure, while performing matching within the matching duration, the processor 230 can also be configured to: extract image features of mouth movements of multiple objects from the matching duration, extract audio features of multiple voices from the matching duration, perform feature matching on the image features and the audio features, and based on the results of the feature matching, match voices corresponding to multiple objects respectively from among the multiple voices.

[0126] According to an embodiment of the present disclosure, when performing feature matching, the processor 230 may also be configured to: measure the similarity between the image feature and the audio feature, wherein a greater weight is assigned to the similarity within a duration in which the image feature changes more over time, and the probability that the multiple voices correspond to the multiple objects is calculated based on the measured similarity. In one embodiment, the processor 230 may also be configured to: measure at least the first similarity and the second similarity between the image feature and the audio feature. The first weight is assigned to the first similarity, and the second weight is assigned to the second similarity. When the first change of the image feature of the first similarity within the first duration is greater than the second change of the image feature of the second similarity within the second duration, the first weight is greater than the second weight.

[0127] According to an embodiment of the present disclosure, when performing feature matching, the processor 230 may also be configured to: perform matching by assigning a greater weight to the image features and audio features in the portion where the image features have a greater change, and calculate the probability that the multiple voices correspond to the multiple objects respectively based on the result of performing the matching. In one embodiment, the processor 230 may also be configured to: perform matching by assigning a first weight to the first image feature and the first audio feature, and assigning a second weight to the second image feature and the second audio feature. When the first change of the first image feature in the first portion is greater than the second change of the second image feature, the first weight is greater than the second weight.

[0128] According to an embodiment of the present disclosure, while extracting image features of mouth movement, the processor 230 may also be configured to: extract a facial area around the mouth of multiple objects as a region of interest (ROI), and extract image features only from the remaining area of ​​the ROI excluding at least a portion of the ROI.

[0129] According to an embodiment of the present disclosure, while extracting image features only from the remaining area, the processor 230 may also be configured to: repeatedly extract image features while differently setting at least a portion of the area excluded from the ROI.

[0130] According to an embodiment of the present disclosure, the processor 230 may also be configured to control the reproduction of the video based on the matching results in the entire duration so that a plurality of voices are output in association with a plurality of objects.

[0131] According to an embodiment of the present disclosure, while controlling the reproduction of the video, the processor 230 may also be configured to: when an input of selecting a first object from multiple objects is received during the reproduction of the video, emphasize the output of a first voice corresponding to the first object among multiple voices.

[0132] According to an embodiment of the present disclosure, an input of selecting a first object from among a plurality of objects may be in the form of zooming in or touching an area where the first object is displayed on a screen where a video is being reproduced.

[0133] According to an embodiment of the present disclosure, while controlling the reproduction of the video, the processor 230 may also be configured to control voices matching the multiple objects to be distributed to the multiple speakers based on the positions of the multiple objects in the video.

[0134] The embodiments of the present disclosure may be implemented or supported by one or more computer programs, and the computer programs may be constructed by computer-readable program codes and recorded in computer-readable media. In the present disclosure, "applications" and "programs" may indicate one or more computer programs, software components, instruction sets, processes, functions, objects, classes, instances, related data, or some of them suitable for implementation in computer-readable program codes. "Computer-readable program codes" may include various types of computer codes, including source code, object code, and executable code. "Computer-readable media" may include various types of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive (HDD), compact disk (CD), or various types of memory.

[0135] Machine-readable storage media may be provided in the form of non-transitory storage media. Here, "non-transitory storage media" are tangible devices, and may exclude wired, wireless, optical, or another communication link that sends transient electrical signals or other signals. "Non-transitory storage media" does not distinguish between situations where data is semi-permanently stored in a storage medium and situations where data is temporarily stored in a storage medium. For example, "non-transitory storage media" may include a buffer that temporarily stores data. Computer-readable media may be any available media that is accessible to a computer, and includes all volatile and non-volatile media and detachable and non-detachable media. Computer-readable media include media that can permanently store data and media that store data and then rewrite it later, such as rewritable optical disks or erasable memory devices.

[0136] According to an embodiment of the present disclosure, the method according to an embodiment of the present disclosure in this specification may be provided by being included in a computer program product. A computer program product is a product that can be traded between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed (e.g., downloaded or uploaded) between two user devices (e.g., smart phones) through an application store or directly or online. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable application) may be at least temporarily generated or temporarily stored in a machine-readable storage medium, such as a memory of a manufacturer's server, an application store's server, or a relay server.

[0137] The above description is provided for illustration, and those of ordinary skill in the art will understand that various changes in form and detail can be easily made therein without departing from the basic characteristics and scope of the present disclosure as defined by the appended claims. For example, even when the above-mentioned techniques are performed in a different order than the described methods, and / or components such as systems, structures, devices, and circuits are combined in a different manner than the described methods, or components are replaced or substituted by another component or an equivalent, appropriate results can be achieved. Therefore, it should be understood that the above-mentioned embodiments of the present disclosure are examples in all respects only and are not limited. For example, each component described as a single type can be implemented in a distributed manner, and similarly, components described as distributed can be implemented in a combined form.

[0138] The scope of the present disclosure is indicated by the appended claims rather than the detailed description above, and it should be interpreted that all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts are included in the scope of the present disclosure.

Claims

1. A method comprising: Separate multiple voices in a video; determining dissimilarity between the plurality of speech sounds; Based on the differences between the multiple voices, select a partial duration within the entire duration of the video as a matching duration; During the matching duration, matching the multiple voices with the multiple objects respectively based on the mouth movements of the multiple objects in the video; as well as Based on the matching results between the multiple voices and the multiple objects within the matching duration, the multiple voices are matched with the multiple objects respectively throughout the entire duration of the video.

2. The method according to claim 1, wherein: The mouth movement includes at least one of a change in lip shapes of the plurality of subjects or a change in facial shapes around mouths of the plurality of subjects.

3. The method according to any one of claims 1 and 2, wherein: Selecting the partial duration in the entire duration of the video as the matching duration includes: dividing the entire duration of the video into a plurality of durations; determining a dissimilarity value between the plurality of speech sounds for each of the plurality of durations; and A duration having a highest calculated dissimilarity value among the plurality of durations is selected as the matching duration.

4. The method according to any one of claims 1 to 3, wherein: Determining the dissimilarity value between the plurality of speech sounds for each of the plurality of durations comprises: determining a standard deviation of the plurality of speech sounds during each of the plurality of durations; and For each of the plurality of durations, a standard deviation among standard deviations of the plurality of voices is determined as the dissimilarity value.

5. The method according to any one of claims 1 to 4, wherein: During the matching duration, matching the multiple voices with the multiple objects in the video respectively includes: extracting image features of mouth movements of the plurality of objects from the matching durations; extracting audio features of the plurality of voices from the matching durations; performing feature matching on the image features and the audio features; and Based on the result of performing the feature matching, voices respectively corresponding to the plurality of objects are matched from among the plurality of voices.

6. The method according to any one of claims 1 to 5, wherein: Performing the feature matching includes measuring at least a first similarity and a second similarity between the image feature and the audio feature, wherein a first weight is assigned to the first similarity and a second weight is assigned to the second similarity, a first change in the image feature over a first duration for the first similarity is greater than a second change in the image feature over a second duration for the second similarity, and the first weight is greater than the second weight, and Based on the measured similarities, probabilities that the plurality of speech sounds respectively correspond to the plurality of objects are determined.

7. The method according to any one of claims 1 to 6, wherein: Executing the feature matching includes: performing matching by assigning a first weight to the first image feature and the first audio feature and assigning a second weight to the second image feature and the second audio feature, wherein a first change in the first image feature in the first portion is greater than a second change in the second image feature, and the first weight is greater than the second weight, and Based on the result of performing the matching, probabilities that the plurality of voices respectively correspond to the plurality of objects are determined.

8. The method according to any one of claims 1 to 7, wherein: Extracting the image features of the mouth movement includes: extracting facial regions including the areas around the mouths of the plurality of subjects as regions of interest (ROIs); and The image features are extracted only from a remaining area of ​​the ROI excluding at least a portion of the area of ​​the ROI.

9. The method according to any one of claims 1 to 8, wherein: Extracting the image feature only from the remaining region includes repeatedly extracting the image feature while differently setting the at least partial region excluded from the ROI.

10. The method according to any one of claims 1 to 9, further comprising: Based on the result of the matching in the entire duration, the reproduction of the video is controlled to output that the plurality of voices are associated with the plurality of objects.

11. A computing device comprising: an input / output interface (220) configured to display a screen on which a video is reproduced and to receive input from a user; A memory (240) for storing instructions; as well as A processor (230), wherein the processor (230) is configured to execute the instructions to: Separating multiple voices in the video; determining dissimilarity between the plurality of speech sounds; Based on the differences between the multiple voices, select a partial duration of the entire duration of the video as a matching duration; During the matching duration, matching the plurality of voices with the plurality of objects respectively based on mouth movements of the plurality of objects in the video; and Based on the matching results between the multiple voices and the multiple objects within the matching duration, the multiple voices are matched with the multiple objects respectively throughout the entire duration of the video.

12. The computing device of claim 11, wherein: The mouth movement includes changes in lip shapes of the plurality of subjects and changes in facial shapes around the mouths of the plurality of subjects.

13. The computing device according to any one of claims 11 and 12, wherein: While selecting the partial duration in the entire duration of the video as the matching duration, the processor (230) is further configured to: dividing the entire duration of the video into a plurality of durations; determining a dissimilarity value between the plurality of speech sounds for each of the plurality of durations; as well as The duration with the highest calculated dissimilarity value is selected as the matching duration.

14. A computing device according to any one of claims 11 to 13, wherein: While determining the dissimilarity value, the processor (230) is further configured to: determining a standard deviation of the plurality of speech sounds for each of the plurality of durations, and For each of the plurality of durations, a standard deviation among standard deviations of the plurality of voices is determined as the dissimilarity value.

15. A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 1 to 10 on a computer.

Images