A video and audio alignment method, device, equipment and storage medium

Abstract

The application discloses a video and audio alignment method and device, equipment and storage medium, wherein the method comprises: acquiring a video and audio alignment instruction, the video and audio alignment instruction being used for aligning at least one target object in a video and audio of the target object; in response to the video and audio alignment instruction, performing encoding processing on at least one frame of image of the video to obtain image features corresponding to the at least one frame of image, wherein the image features comprise target image features of at least one target object in the video; performing encoding processing on audio corresponding to the at least one frame of image to obtain audio features corresponding to the at least one frame of image, wherein the audio features comprise target audio features of the at least one target object; and performing similarity comparison on at least one target image feature and at least one target audio feature to obtain target audio corresponding to the at least one target object in the image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and includes, but is not limited to, a method, apparatus, device, and storage medium for audio-visual alignment. Background Technology

[0002] When playing video files, users cannot extract the corresponding audio from a single target in the video frame, including people, animals, and physical devices that make sounds (television, musical instruments, etc.). Summary of the Invention

[0003] In view of this, embodiments of this application provide an audio-visual correspondence method, apparatus, device, and storage medium.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] In a first aspect, embodiments of this application provide an audio-visual alignment method, the method comprising:

[0006] Obtain audio-visual alignment instructions, which are used to align at least one target object in the video with the audio of the target object;

[0007] In response to the audio-visual alignment instruction, at least one frame of the video is encoded to obtain image features corresponding to the at least one frame, wherein the image features include target image features of at least one target object in the video;

[0008] The audio corresponding to the at least one frame of image is encoded to obtain the audio features corresponding to the at least one frame of image, wherein the audio features include the target audio features of the at least one target object;

[0009] A similarity comparison is performed on at least one of the target image features and at least one of the target audio features to obtain the target audio corresponding to the image that is missing one of the target objects.

[0010] Secondly, embodiments of this application provide an audio-visual alignment device, the device comprising:

[0011] The acquisition module is used to acquire audio-visual alignment instructions, which are used to align at least one target object in the video with the audio of the target object;

[0012] The first encoding module is configured to encode at least one frame of the video in response to the audio-visual alignment instruction to obtain image features corresponding to the at least one frame, wherein the image features include target image features of at least one target object in the video.

[0013] The second encoding module is used to encode the audio corresponding to the at least one frame of image to obtain the audio features corresponding to the at least one frame of image, wherein the audio features include the target audio features of the at least one target object;

[0014] The comparison module is used to perform similarity comparison on at least one of the target image features and at least one of the target audio features to obtain the target audio corresponding to the image that is missing one of the target objects.

[0015] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement the above-described method.

[0016] Fourthly, embodiments of this application provide a storage medium storing executable instructions for implementing the above-described method when executed by a processor.

[0017] In this embodiment, firstly, an audio-visual alignment instruction is obtained, which is used to align the audio of at least one target object in the video with the audio of the target object; then, in response to the audio-visual alignment instruction, at least one frame of the video is encoded to obtain image features corresponding to the at least one frame, the image features including target image features of at least one target object in the video; the audio corresponding to the at least one frame is encoded to obtain audio features corresponding to the at least one frame, the audio features including target audio features of the at least one target object; finally, a similarity comparison is performed between at least one target image feature and at least one target audio feature to obtain the target audio corresponding to at least one target object in the image. In this way, the audio corresponding to at least one target object in the video can be extracted. Attached Figure Description

[0018] Figure 1A A schematic diagram illustrating the implementation process of an audio-visual alignment method provided in this application embodiment;

[0019] Figure 1B A schematic diagram of a frame of image provided in an embodiment of this application;

[0020] Figure 1C A schematic diagram of an audio encoding method provided in an embodiment of this application;

[0021] Figure 2 A schematic diagram illustrating the implementation process of an audio-visual alignment method provided in this application embodiment;

[0022] Figure 3A A schematic diagram illustrating the implementation process of an audio-visual alignment method provided in this application embodiment;

[0023] Figure 3B A schematic diagram of an audio-visual operation screen provided in an embodiment of this application;

[0024] Figure 4A This is a schematic diagram of the structure of an audio extraction network provided in an embodiment of this application;

[0025] Figure 4B This is a schematic diagram of the structure of a Transformer decoder provided in an embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the composition structure of an audio-visual alignment device provided in an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of the embodiments will be further described in detail below with reference to the accompanying drawings. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0029] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0030] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0032] This application provides an audio-visual alignment method, such as... Figure 1A As shown, the method includes:

[0033] Step S110: Obtain audio-visual alignment instructions, wherein the audio-visual alignment instructions are used to align at least one target object in the video with the audio of the target object;

[0034] In some embodiments, obtaining the audio-visual alignment instruction can utilize the user's visual information as the instruction. For example, the user's gaze focus can be obtained to determine if that gaze focus is on a target object in the video. Once it is determined that the user is focusing on the target object, audio-visual alignment is initiated.

[0035] In some embodiments, the system can receive a user's instruction to click on a target object in the video and initiate audio-visual alignment; it can also receive a user's instruction to circle and click on a target object in the video and initiate audio-visual alignment; it can also determine a nearby target object and initiate audio-visual alignment when the user clicks or circles a position in the video; or it can receive a user's instruction to point to a target object in the video and initiate audio-visual alignment.

[0036] During implementation, there are no restrictions on how users select target objects in the video.

[0037] Step S120: In response to the audio-visual alignment instruction, at least one frame of the video is encoded to obtain image features corresponding to the at least one frame, wherein the image features include target image features of at least one target object in the video;

[0038] During implementation, a convolutional neural network encoder (CNN Encoder) can be used to encode at least one frame of the video to obtain the image features corresponding to at least one frame.

[0039] Step S130: Encode the audio corresponding to the at least one frame of image to obtain the audio features corresponding to the at least one frame of image, wherein the audio features include the target audio features of the at least one target object;

[0040] During implementation, a Long Short-Term Memory (LSTM) encoder can be used to encode the audio corresponding to at least one frame of an image to obtain the audio features corresponding to at least one frame of an image.

[0041] Step S140: Perform a similarity comparison on at least one of the target image features and at least one of the target audio features to obtain the target audio corresponding to the image that is missing one of the target objects.

[0042] During implementation, the at least one target image feature can be decoded to obtain image features corresponding to each target object; the at least one target audio feature can be decoded and filtered to obtain audio features corresponding to each target object; the image features corresponding to each target object and the audio features corresponding to each target object can be compared to obtain the target audio corresponding to each target object in the image.

[0043] Here, the image features corresponding to each target object are compared with the audio features corresponding to each target object. A fully connected layer can be used to perform feature matching on the image features corresponding to each target object and the audio features corresponding to each target object to obtain the target audio corresponding to each target object in the image.

[0044] In some embodiments, a Transformer structure can be used for feature matching, i.e., similarity comparison. The target audio features are decoded by an audio network and then filtered through multiple channels to distinguish different audio information. This yields different audio segments, but not yet the target object corresponding to each audio segment. The target image features are decoded by a vision network to obtain a segmentation mask. Different segmented regions are processed through fully connected layers to obtain corresponding feature representations, which are then matched with the audio feature information output from the multi-channel filtering. This establishes connections between the target object and different target audio segments, thereby obtaining the target audio corresponding to each target object.

[0045] Figure 1B A schematic diagram of a frame of image provided in an embodiment of this application, as shown below. Figure 1B As shown, the image includes a man 11, a woman 12, a horse 13, and a dog 14, among which,

[0046] During implementation, the image frame is encoded to obtain its corresponding image features, including the image features of man 11, woman 12, horse 13, and dog 14. The corresponding audio is also encoded to obtain its corresponding audio features, including the audio features of man 11, woman 12, horse 13, and dog 14. A similarity comparison is performed between the image features and audio features of man 11, woman 12, horse 13, and dog 14 in the image to obtain the audio corresponding to man 11, woman 12, horse 13, and dog 14.

[0047] In this embodiment, firstly, an audio-visual alignment instruction is obtained, which is used to align the audio of at least one target object in the video with the audio of the target object; then, in response to the audio-visual alignment instruction, at least one frame of the video is encoded to obtain image features corresponding to the at least one frame, the image features including target image features of at least one target object in the video; the audio corresponding to the at least one frame is encoded to obtain audio features corresponding to the at least one frame, the audio features including target audio features of the at least one target object; finally, a similarity comparison is performed between at least one target image feature and at least one target audio feature to obtain the target audio corresponding to at least one target object in the image. In this way, the audio corresponding to at least one target object in the video can be extracted.

[0048] In some embodiments, the step S120 above, "encoding at least one frame of the video to obtain image features corresponding to the at least one frame," can be achieved through the following steps:

[0049] Step S121: Obtain at least one frame of the video image and a video mask corresponding to each frame of the video image. The video mask divides the video image into at least one target region, and each target region corresponds to a target object.

[0050] During implementation, the phase coherence data of each pixel in the video image is binarized and thresholded to obtain the video mask corresponding to the image. Here, the video mask is a segmented image including the target object.

[0051] Step S122: Perform convolutional coding on at least one of the video frames and the video mask corresponding to each of the video frames to obtain the image features corresponding to at least one frame.

[0052] During implementation, the video image and its corresponding video mask are simultaneously input into a convolutional neural network encoder to obtain the image features corresponding to the video image. Here, the image features include the target image features of at least one target object in the video, which can be an object selected by the user.

[0053] In this embodiment, at least one frame of the video and a video mask corresponding to each frame are first obtained; then, convolutional coding is performed on the at least one frame of the video and the video mask corresponding to each frame. This effectively obtains image features including the target object.

[0054] In some implementations, the step S130 above, "encoding the audio corresponding to the at least one frame of image to obtain the audio features corresponding to the at least one frame of image," can be achieved through the following steps:

[0055] Step 131: Obtain the Mel frequency cepstral coefficients (MFCCs) of the audio.

[0056] During implementation, the raw audio data can be preprocessed to obtain MFCC sequences, as follows:

[0057] 1. Pre-emphasize, frame, and window the original speech;

[0058] 2. For each short-time analysis window, the corresponding spectrum is obtained by using Fast Fourier Transform (FFT);

[0059] 3. Obtain the Mel spectrum by passing the spectrum obtained through FFT through the Mel filter bank;

[0060] 4. Perform cepstral analysis on the Mel frequency spectrum to obtain the Mel frequency cepstral coefficients (MFCCs);

[0061] Here, cepstral analysis includes taking the logarithm and performing the inverse transform. The actual inverse transform can be achieved through the Discrete Cosine Transform (DCT). For example, the 2nd to 13th coefficients after the DCT can be taken as MFCC coefficients.

[0062] After completing steps 1 to 4 above, the Mel frequency cepstral coefficients of the audio can be obtained, which are the required preprocessing features.

[0063] Figure 1C This is a schematic diagram of an audio encoding method provided in an embodiment of this application, such as... Figure 1C As shown in the diagram, the audio data obtained after data preprocessing 15 and the audio data obtained after passing through an N-layer LSTM structure 16 are included.

[0064] Here, as Figure 1B As shown, the Mel frequency cepstral coefficients of the audio can be obtained after data preprocessing 15.

[0065] Step 132: Normalize the MFCC to obtain the normalized features of the audio.

[0066] During implementation, the obtained MFCC sequence can be normalized. By performing mean normalization on each dimension, channel compensation can be achieved, eliminating overfitting or slow training caused by factors such as amplitude and interference in different sound signals.

[0067] Step 133: Perform long short memory processing on the normalized features to obtain the audio features corresponding to the at least one frame of image.

[0068] During implementation, such as Figure 1B As shown, normalized features can be fed into a long short-term memory network encoder for feature learning. Through an N-layer LSTM structure, encoded features are obtained, where N is an integer greater than or equal to 1.

[0069] In this embodiment, the Mel-frequency cepstral coefficients (MFCCs) of the audio are first obtained; then, the MFCCs are normalized to obtain the normalized features of the audio; finally, the normalized features are processed using long short-term memory (LSM) to obtain the audio features corresponding to at least one frame of the image. In this way, the audio features corresponding to the image can be obtained.

[0070] In some embodiments, step S140 above, "comparing at least one of the target image features and at least one of the target audio features to obtain the target audio corresponding to the image missing one of the target objects," is as follows: Figure 2 As shown, this can be achieved through the following steps:

[0071] Step S210: Based on the image features and at least one of the target image features, determine the depth information of at least one target object in the at least one frame image;

[0072] Depth of field (DOF) refers to the range of distances in front of and behind an object in an image that allows for a sharp image to be captured by the camera lens or other imager. In other words, it's the relative position of each object within its environment. For example, a large object in the foreground of an image will have a different depth of field than a small object in the background.

[0073] During implementation, the depth information of the target object in the whole image can be determined based on the background features of the whole image and the target image features of the background area. For example, the depth information of the target object located in the middle foreground background and the depth information of the target object located in the left and right rear background can be determined respectively.

[0074] Step S220: Based on the audio features and at least one of the target audio features, determine the time delay information of at least one of the target audio features;

[0075] During implementation, since the audio latency of objects at different depths of field in the video varies when it reaches the user's location, it is necessary to determine the latency of the target audio one by one.

[0076] Step S230: Match the depth information with the time delay information to obtain at least one set of target objects and target audio located at the same depth in the image;

[0077] Here, target objects located at the same depth of field can be grouped together; then target audio with the same time delay information can be grouped together; finally, by using the values ​​of depth of field and time delay, the grouped target objects and target audio can be matched to obtain at least one group of target objects and target audio located at the same depth of field in the image.

[0078] Step S240: Identify the target objects and target audio located at the same depth of field, and obtain the target audio corresponding to at least one of the target objects in the depth of field.

[0079] During implementation, since there may be multiple target objects and target audios located at the same depth of field, the target object and target audio can be matched based on the characteristics of the target object and the target audio to obtain the target audio corresponding to the target object.

[0080] In this embodiment, firstly, based on the image features and at least one target image feature, the depth information of at least one target object in the at least one frame of image is determined; based on the audio features and at least one target audio feature, the time delay information of at least one target audio is determined; then, the depth information and the time delay information are matched to obtain at least one set of target objects and target audio located at the same depth of field in the image; finally, the target objects and target audio located at the same depth of field are identified to obtain the target audio corresponding to at least one target object in the depth of field. In this way, target objects and target audio can be classified based on depth of field first, and then the target audio corresponding to the target object can be obtained by identifying target objects and target audio located at the same depth of field, effectively improving the efficiency and accuracy of identification.

[0081] In some embodiments, the above step S240, "identifying the target objects and target audio located at the same depth of field, and obtaining the target audio corresponding to at least one of the target objects in the depth of field," can be achieved through the following steps:

[0082] Step 241: Identify the target image features corresponding to the target objects located at the same depth of field to determine the image attribute information of the target objects;

[0083] The image attribute information includes at least: type, age, and gender.

[0084] For example, it can identify target objects as men, women, children, dogs, cats, etc.

[0085] Step 242: Identify the target audio features corresponding to the target audio located at the same depth of field to determine the sound attribute information of the target audio;

[0086] The sound attribute information includes at least: timbre and pitch.

[0087] Step 243: Match the image attribute information and the sound attribute information to obtain the target audio corresponding to at least one of the target objects in the depth of field.

[0088] For example, a man's timbre and pitch are different from a woman's. Through matching, men can be matched with men's audio, and women can be matched with women's audio.

[0089] In this embodiment, firstly, target image features corresponding to target objects located at the same depth of field are identified to determine the image attribute information of the target objects; then, target audio features corresponding to target audio located at the same depth of field are identified to determine the sound attribute information of the target audio; finally, the image attribute information and the sound attribute information are matched to obtain the target audio corresponding to at least one target object in the depth of field. In this way, it is possible to match the audio most similar to the sound attribute information of different target objects in the same depth of field.

[0090] In some embodiments, such as Figure 3A As shown, the method for aligning audio and video also includes the following steps:

[0091] Step S150: In response to the instruction to select any target object in the image, adjust the target audio corresponding to the selected target object.

[0092] During implementation, users can be provided with multiple interaction methods to select video targets, including clicking, selecting all, and selecting the nearest target.

[0093] Figure 3B This is a schematic diagram of an audio-visual operation screen provided in an embodiment of this application, such as... Figure 3B As shown, the diagram includes figure 31, television 32, figure 33, and figure 34, among which,

[0094] During implementation, the volume of character 31 can be adjusted individually in response to selecting character 31; the corresponding volume can be adjusted in response to selecting both character 31 and character 33 simultaneously; and the volume of television 32 can be turned down in response to selecting television 32.

[0095] In this way, users can select a single target object through the video and block out the sound of other target objects.

[0096] Step S160: In response to the instruction to select any position in the image, adjust the target audio corresponding to all target objects in the image.

[0097] like Figure 3B As shown, the schematic diagram also includes position 35 and position 35, wherein,

[0098] The audio volume is loud and comes from the right side when the click position is 35, the character is 33, and the sound is loud.

[0099] It can respond to a click on position 36, with the TV audio at position 35 louder from the right, the other parts quieter, and human voices coming from the left.

[0100] In this way, users can select a specific location in the video, and the system will automatically find surrounding target objects and extract the target audio corresponding to each target object to generate new ambient sound.

[0101] In this embodiment, in response to an instruction to select any target object in the image, the target audio corresponding to the selected target object is adjusted. In response to an instruction to select any position in the image, the target audio corresponding to all target objects in the image is adjusted. Thus, multiple methods for adjusting, selecting, and combining audio for independent target objects are provided simultaneously, including adjusting the volume of a single target object, and reconstructing the audio output based on the nearest target object according to the selected target object's position in the image.

[0102] Figure 4A This is a schematic diagram of the structure of an audio extraction network provided in an embodiment of this application, as shown below. Figure 4A As shown, the audio extraction network includes: a convolutional neural network encoder 44, a long short-term memory network encoder 45, and a Transformer decoder 48, wherein,

[0103] In some embodiments, complete image information of the video, segmented image information of a specific target, and complete audio information can be used as input to the network, and the audio information of the specific target can be used as the ground truth (also the output) to construct a multimodal deep learning network for training, generating, for example,... Figure 4A The illustrated audio extraction network can select based on specific target visual information, thereby aligning any target within a video with its generated audio. Here, in machine learning, the term "ground truth" refers to the accuracy of the training set in classifying supervised learning techniques.

[0104] During implementation, the audio extraction network can be used based on the following steps:

[0105] Step 410: Input the complete image information 42 and the segmented image information 43 of the specific target into the convolutional neural network encoder 44 to obtain image features 46;

[0106] Step 420: The complete audio information (Melp sequence) 41 is passed through a long short-term memory network encoder 45 to obtain audio features 47;

[0107] Step 430: Input the image features and audio features into the Transformer decoder 48 to obtain the audio Melp sequence 49 of the specific target.

[0108] In this embodiment, complete image information of the video, segmented image information of a specific target, and complete audio information are used as input to the audio extraction network, so that any target in the video input to the audio extraction network can be aligned with its generated audio.

[0109] Figure 4B A schematic diagram of the structure of a Transformer decoder 48 provided in an embodiment of this application is shown below. Figure 4B As shown, the Transformer decoder 48 includes: a video sub-network 481, an audio sub-network 482, multi-channel filtering 483, and feature matching 484, wherein,

[0110] During implementation, the Transformer decoder 48 can decode the audio and image time series features and perform similarity comparison based on the image features and audio features extracted by the convolutional neural network encoder and the long short-term memory network encoder, respectively. The audio and the corresponding target image are matched by the distance between different features.

[0111] For image features and audio features, such as Figure 4B As shown, a Transformer architecture is used for feature matching. Audio features 47 are decoded by an audio network 482 and then filtered by a multi-channel filter 483 to distinguish different audio information. Different audio segments are obtained, but the target object corresponding to each audio segment is not yet determined. Image features 46 are decoded by a video network 481 to obtain a segmentation mask. Different segmented regions are then matched by feature matching 484 to obtain corresponding feature representations. These representations are then matched with the audio feature information output by the multi-channel filter 483, establishing connections between the target object and different audio information, thus obtaining the audio information corresponding to different target objects. Here, feature matching 484 can be implemented using fully connected layers (FC).

[0112] During training, for matching image features and audio features, an audiovisual matching loss function L can be used. AM This constrains the sound-producing objects and audio features to maintain a similar distribution in the feature space.

[0113] Loss function L AM Define the following formula (1):

[0114]

[0115] Where KL is the Kullback-Leibler divergence, used to calculate the magnitude of the loss function, M is the image visual features, A is the audio features, and i represents the i-th target, with a total of n targets. Entropy To minimize the cross-entropy loss function based on KL divergence for each object, as shown in Equation (2):

[0116]

[0117] Where, m i Image features for each target, a j For the audio features of each target, y ij This represents the labeled information and whether the features of the two match.

[0118] Based on the foregoing embodiments, this application provides an audio-visual alignment device. The device includes various modules, each module includes sub-modules, and each sub-module includes units. It can be implemented by a processor in an electronic device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0119] Figure 5 This is a schematic diagram of the composition structure of the audio-visual alignment device provided in the embodiments of this application, as shown below. Figure 5 As shown, the device 500 includes:

[0120] The acquisition module 510 is used to acquire audio-visual alignment instructions, which are used to align at least one target object in the video with the audio of the target object;

[0121] The first encoding module 520 is configured to encode at least one frame of the video in response to the audio-visual alignment instruction to obtain image features corresponding to the at least one frame, wherein the image features include target image features of at least one target object in the video.

[0122] The second encoding module 530 is used to encode the audio corresponding to the at least one frame of image to obtain the audio features corresponding to the at least one frame of image, wherein the audio features include the target audio features of the at least one target object;

[0123] The comparison module 540 is used to perform a similarity comparison on at least one of the target image features and at least one of the target audio features to obtain the target audio corresponding to the image that is missing one of the target objects.

[0124] In some embodiments, the first encoding module 520 includes a first acquisition submodule and a convolutional encoding submodule, wherein the first acquisition submodule is used to acquire at least one frame of video image and a video mask corresponding to each frame of video image, the video mask dividing the video image into at least one target region, each target region corresponding to a target object; the convolutional encoding submodule is used to perform convolutional encoding processing on at least one frame of video image and the video mask corresponding to each frame of video image to obtain image features corresponding to at least one frame of image.

[0125] In some embodiments, the second encoding module 530 includes a second acquisition submodule, a normalization submodule, and a long short-term memory submodule, wherein the second acquisition submodule is used to acquire the Mel frequency cepstral coefficients (MFCCs) of the audio; the normalization submodule is used to normalize the MFCCs to obtain normalized features of the audio; and the long short-term memory submodule is used to perform long short-term memory processing on the normalized features to obtain audio features corresponding to the at least one frame of image.

[0126] In some embodiments, the comparison module 540 includes a first determining submodule, a second determining submodule, a matching submodule, and an identification submodule. The first determining submodule is configured to determine the depth information of at least one target object in the at least one frame of an image based on the image features and at least one target image feature. The second determining submodule is configured to determine the time delay information of at least one target audio based on the audio features and at least one target audio feature. The matching submodule is configured to match the depth information with the time delay information to obtain at least one set of target objects and target audio located at the same depth of field in the image. The identification submodule is configured to identify the target objects and target audio located at the same depth of field to obtain the target audio corresponding to at least one target object in the depth of field.

[0127] In some embodiments, the recognition submodule includes a first recognition unit, a second recognition unit, and a matching unit, wherein the first recognition unit is used to recognize target image features corresponding to the target objects located at the same depth of field to determine the image attribute information of the target objects; the second recognition unit is used to recognize target audio features corresponding to the target audio located at the same depth of field to determine the sound attribute information of the target audio; and the matching unit is used to match the image attribute information and the sound attribute information to obtain the target audio corresponding to at least one of the target objects in the depth of field.

[0128] In some embodiments, the image attribute information includes at least: type, age, and gender; the sound attribute information includes at least: timbre and pitch.

[0129] In some embodiments, the apparatus includes a first adjustment module for adjusting the target audio corresponding to all target objects in the image in response to an instruction to select any position in the image.

[0130] In some embodiments, the apparatus includes a second adjustment module for adjusting the target audio corresponding to all target objects in the image in response to an instruction to select any position in the image.

[0131] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0132] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of software products. These computer software products are stored in a storage medium and include several instructions to cause electronic devices (such as mobile phones, tablets, laptops, desktop computers, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0133] Correspondingly, this application provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the audio-visual alignment method provided in the above embodiments.

[0134] Correspondingly, embodiments of this application provide an electronic device, Figure 6 A schematic diagram of a hardware entity of an electronic device provided in an embodiment of this application, such as... Figure 6 As shown, the hardware entity of the device 600 includes a memory 601 and a processor 602. The memory 601 stores a computer program that can run on the processor 602. When the processor 602 executes the program, it implements the steps in the audio-visual alignment method provided in the above embodiments.

[0135] The memory 601 is configured to store instructions and applications executable by the processor 602, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) of the processor 602 and various modules in the electronic device 600, and can be implemented by flash memory or random access memory (RAM).

[0136] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0137] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0138] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0140] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0141] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0142] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0143] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a mobile phone, tablet computer, laptop computer, desktop computer, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0144] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0145] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0146] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0147] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for aligning audio and video, the method comprising: Obtain audio-visual alignment instructions, which are used to align at least one target object in the video with the audio of the target object; In response to the audio-visual alignment instruction, at least one frame of the video image and a video mask corresponding to each frame of the video image are obtained. The video mask is used to divide the video image into at least one target region, and each target region corresponds to a target object. Convolutional coding is performed on at least one of the video frames and the video mask corresponding to each of the video frames to obtain image features corresponding to at least one frame. The image features include target image features of at least one target object in the video. The target image features include at least the depth information of the target object in at least one frame. The audio corresponding to the at least one frame of image is encoded to obtain the audio features corresponding to the at least one frame of image, wherein the audio features include the target audio features of the at least one target object; When the target objects have the same depth information in at least one frame of image, the target audio features corresponding to the target audio located in the same depth are identified, and the sound attribute information of the target audio features is matched. The image attribute information and the sound attribute information are matched to obtain the target audio corresponding to at least one target object in the depth. A similarity comparison is performed on at least one of the target image features and at least one of the target audio features to obtain the target audio corresponding to at least one of the target objects in the image.

2. The method as described in claim 1, wherein encoding the audio corresponding to the at least one frame of image to obtain the audio features corresponding to the at least one frame of image includes: Obtain the Mel-frequency cepstral coefficients (MFCCs) of the audio. The MFCC is normalized to obtain the normalized features of the audio. The normalized features are processed using long short-term memory to obtain the audio features corresponding to the at least one frame of the image.

3. The method as described in claim 1, wherein performing a similarity comparison on at least one target image feature and at least one target audio feature to obtain the target audio corresponding to at least one target object in the image comprises: Based on the image features and at least one of the target image features, determine the depth information of at least one target object in the at least one frame of image; Based on the audio features and at least one of the target audio features, determine the time delay information of at least one of the target audio features; By matching the depth information with the time delay information, at least one set of target objects and target audio located at the same depth in the image is obtained; Identify the target objects and target audio located at the same depth of field, and obtain the target audio corresponding to at least one of the target objects in the depth of field.

4. The method as described in claim 1, wherein the image attribute information includes at least: Type, age, or gender; The sound attribute information includes at least: timbre and pitch.

5. The method according to any one of claims 1 to 4, wherein the method further comprises: In response to an instruction to select any target object in the image, the target audio corresponding to the selected target object is adjusted.

6. The method according to any one of claims 1 to 4, wherein the method further comprises: In response to an instruction to select any location in the image, adjust the target audio corresponding to all target objects in the image.

7. An audio-visual alignment device, the device comprising: The acquisition module is used to acquire audio-visual alignment instructions, which are used to align at least one target object in the video with the audio of the target object; The first encoding module is used to respond to the audio-visual alignment instruction to obtain at least one frame of video image and a video mask corresponding to each frame of video image. The video mask is used to divide the video image into at least one target region, and each target region corresponds to a target object. The second encoding module is configured to perform convolutional encoding processing on at least one of the video frames and the video mask corresponding to each of the video frames to obtain image features corresponding to at least one frame, wherein the image features include target image features of at least one target object in the video, and the target image features include at least the depth information of the target object in at least one frame; the module is also configured to encode the audio corresponding to the at least one frame to obtain audio features corresponding to the at least one frame, wherein the audio features include target audio features of the at least one target object; When the depth information of the target objects is the same in at least one frame of image, the target audio features corresponding to the target audio located in the same depth of field are identified, and the sound attribute information of the target audio features is matched. The image attribute information and the sound attribute information are matched to obtain the target audio corresponding to at least one target object in the depth of field. The comparison module is used to perform similarity comparison on at least one of the target image features and at least one of the target audio features to obtain the target audio corresponding to at least one of the target objects in the image.

8. An electronic device comprising a memory and a processor, the memory storing a computer program that runs on the processor, the processor executing the program to implement the steps of the method according to any one of claims 1 to 6.

Images