Video processing method and device, computer device and readable storage medium

Abstract

The present disclosure provides a video processing method and device, computer equipment and a readable storage medium, wherein the method comprises: determining a target video frame image in a to-be-processed video and a target audio signal corresponding to the target video frame image; performing first encoding processing on the target video frame image to obtain image feature data; and performing second encoding processing on the target audio signal to obtain audio feature data; performing feature fusion processing on the image feature data and the audio feature data to obtain fusion feature data; and performing decoding processing based on the fusion feature data to obtain first position information of a pixel point of a sound-producing object corresponding to the audio signal in the target video frame image in the target video frame image. The present embodiment can improve the positioning accuracy of the sound-producing object in the to-be-processed video.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a video processing method, apparatus, computer equipment, and readable storage medium. Background Art

[0002] Visual signals and audio signals are two extremely important signals received by human senses. The two complement each other to help human senses better recognize and feel the surrounding things. For example, when enjoying a concert, you can not only see the various instruments being played by the performers (i.e., visual signals), but you can also hear the sounds made by the instruments (i.e., audio signals). More importantly, when the sounds of multiple instruments are mixed together, human senses can also distinguish which sound corresponds to which instrument. Audio signals and visual signals describe the same sound object from two aspects, so the above characteristics can be used to carry out research on audio-visual scenes.

[0003] Most existing technologies focus on the problem of sound source localization, that is, locating the sound-making object in a video through audio and video signals. However, they can only achieve rough positioning of large areas, and the positioning accuracy of the sound-making object is low. Summary of the Invention

[0004] The embodiments of the present disclosure at least provide a video processing method, apparatus, computer equipment, and readable storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a video processing method, including:

[0006] Determining a target video frame image in a video to be processed and a target audio signal corresponding to the target video frame image;

[0007] Performing a first encoding process on the target video frame image to obtain image feature data; and performing a second encoding process on the target audio signal to obtain audio feature data;

[0008] Performing feature fusion processing on the image feature data and the audio feature data to obtain fused feature data;

[0009] Decoding is performed based on the fused feature data to obtain first position information of a pixel point of a sound-emitting object corresponding to the audio signal in the target video frame image.

[0010] The embodiment of the present disclosure can perform encoding-decoding processing based on image feature data and audio feature data, and obtain the first position information of the pixel point of the sound object corresponding to the audio signal in the target video frame image at the pixel level, thereby improving the positioning accuracy of the sound object in the processed video.

[0011] In an optional implementation, determining a target video frame image in the video to be processed and a target audio signal corresponding to the target video frame image includes:

[0012] Acquire the video to be processed and the audio signal to be processed corresponding to the video to be processed, and divide the video to be processed into a plurality of sub-videos to be processed;

[0013] According to the division method corresponding to the sub-video to be processed, the audio signal to be processed is divided into target audio signals corresponding to multiple sub-videos to be processed, and the multiple sub-videos to be processed are respectively subjected to frame extraction processing to obtain target video frame images corresponding to multiple sub-videos to be processed.

[0014] In this way, the corresponding target video frame image and target audio signal can be obtained according to business needs.

[0015] In an optional implementation, performing a first encoding process on the target video frame image to obtain image feature data includes:

[0016] Performing multi-level feature extraction processing on the target video frame image to obtain intermediate image feature data corresponding to the multi-level feature extraction processing;

[0017] For each level of feature extraction in the multi-level feature extraction, the intermediate image feature data corresponding to each level of feature extraction processing is subjected to parallel sampling of multiple sampling rates of hole convolution to obtain the image feature data corresponding to the target video frame image.

[0018] In this way, the granularity of feature extraction can be increased.

[0019] In an optional implementation, performing a second encoding process on the target audio signal to obtain audio feature data includes:

[0020] Converting the target audio signal into frequency spectrum data;

[0021] Perform feature extraction processing on the frequency spectrum data to obtain the audio feature data.

[0022] In an optional implementation, performing feature fusion processing on the image feature data and the audio feature data to obtain fused feature data includes:

[0023] Performing size conversion processing on the audio feature data to obtain converted audio feature data whose size matches the image feature data;

[0024] determining a similarity between the converted audio feature data and the image feature data;

[0025] Based on the similarity, the image feature data is adjusted to obtain the fused feature data.

[0026] In this way, the image feature data can be fused based on the audio feature data to give more attention to the sound-making object, making it easier to identify the sound-making object.

[0027] In an optional embodiment, performing size conversion processing on the audio feature data to obtain converted audio feature data whose size matches the image feature data includes:

[0028] performing linear transformation processing on the audio feature data based on the number of data channels of the image feature data to obtain intermediate audio feature data whose number of data channels is consistent with that of the image feature data;

[0029] Based on the height and width of the image feature data, the intermediate audio feature data is spatially repeated to obtain the converted audio feature data.

[0030] In this way, the format of the audio feature data is converted to facilitate the fusion process.

[0031] In an optional implementation, determining the similarity between the converted audio feature data and the image feature data includes:

[0032] Performing convolution processing on the image feature data using a first convolution kernel to obtain first convolution image feature data;

[0033] and, performing convolution processing on the converted audio feature data using a second convolution kernel to obtain first convolution-converted audio feature data;

[0034] Performing dimensionality reduction processing on the first convolution image feature data and the first convolution converted audio feature data respectively to obtain second convolution image feature data and second convolution converted audio feature data;

[0035] Performing a dot product process on the second convolution image feature data and the second convolution-converted audio feature data to obtain the similarity.

[0036] In an optional implementation, adjusting the image feature data based on the similarity to obtain the fused feature data includes:

[0037] Performing convolution processing on the image feature data using a third convolution kernel to obtain third convolution image feature data, and performing dimensionality reduction processing on the third convolution image feature data to obtain fourth convolution image feature data;

[0038] Performing a dot product process on the fourth convolution image feature data and the similarity to obtain adjusted feature data corresponding to the image feature data;

[0039] The adjustment feature data and the image feature data are fused to obtain the fused feature data.

[0040] In an optional implementation, fusing the adjustment feature data and the image feature data to obtain the fused feature data includes:

[0041] Performing dimensionality upscaling on the adjusted feature data to obtain adjusted feature data after dimensionality upscaling; wherein the data dimension of the adjusted feature data after dimensionality upscaling is the same as the data dimension of the image feature data;

[0042] Using a fourth convolution kernel to perform convolution processing on the dimension-upgraded adjusted feature data to obtain target adjusted feature data;

[0043] The target adjustment feature data and the image feature data are superimposed to obtain the fusion feature data.

[0044] In an optional embodiment, the multi-level feature extraction process includes: a final-level feature extraction process, and a non-final-level feature extraction process; the fused feature data includes: first fused feature data corresponding to the final-level feature extraction process, and second fused feature data corresponding to the non-final-level feature extraction process;

[0045] The decoding process based on the fused feature data to obtain first position information of the sound object corresponding to the audio signal in the target video frame image includes:

[0046] Upsampling the first fused feature data corresponding to the final-level feature extraction process to obtain decoded feature data corresponding to the final-level feature extraction process; and

[0047] fusing the second fused feature data corresponding to the non-final-level feature extraction process and the decoded feature data corresponding to the next-level feature extraction process corresponding to the non-final-level feature extraction to obtain fused feature data corresponding to the non-final-level feature extraction process, and upsampling the fused feature data corresponding to the non-final-level feature extraction process to obtain decoded feature data corresponding to the non-final-level feature extraction process;

[0048] Based on the decoded feature data corresponding to the first-level feature extraction processing in the non-final-level feature extraction processing, the first position information of the pixel point of the sound object corresponding to the audio signal in the target video frame image is obtained.

[0049] In this way, through the decoding process, the first position information of the pixel point of the sound-emitting object in the target video frame image can be output, thereby improving the positioning accuracy of the sound-emitting object in the video to be processed.

[0050] In an optional embodiment, the method further includes:

[0051] Based on first position information of a pixel point of a sound-emitting object corresponding to the audio signal in the target video frame image, a mask video frame image including a mask corresponding to the sound-emitting object is generated.

[0052] In an optional embodiment, the video processing method is applied to a pre-trained target neural network, wherein the target neural network includes: an encoder network and a decoder network;

[0053] The encoder network is used to perform a first encoding process on the target video frame image to obtain image feature data; and perform a second encoding process on the target audio signal to obtain audio feature data; and perform feature fusion processing on the image feature data and the audio feature data to obtain fused feature data;

[0054] The decoder network is used to perform decoding processing based on the fused feature data to obtain first position information of a pixel point of a sound-emitting object corresponding to the audio signal in the target video frame image.

[0055] In this way, the target neural network can be trained.

[0056] In an optional embodiment, the method further includes:

[0057] Acquire sample data; the sample data includes first sample data for performing semi-supervised training on the neural network, and / or second sample data for performing fully supervised training on the neural network;

[0058] The neural network to be trained is trained using the sample data to obtain the target neural network.

[0059] In this way, the neural network can be trained based on sample data to obtain a neural network model with higher output accuracy.

[0060] In an optional embodiment, the sample data includes the first sample data, and the first sample data includes: multiple frames of first sample images, first sample audio signals corresponding to each frame of the first sample image, and annotation information corresponding to the first frame of the first sample image; the annotation information is used to indicate first position information of a pixel point of a single sounding target object in the first sample image;

[0061] The obtaining of sample data includes:

[0062] Acquire a first original sample video and a first original sample audio signal corresponding to the first original sample video, and divide the first original sample video into a plurality of first sample sub-videos;

[0063] Dividing the first original sample audio signal into first sample audio signals corresponding to a plurality of first sample sub-videos, respectively, according to a division method corresponding to the first sample sub-video, and performing frame extraction processing on the plurality of first sample sub-videos to obtain first sample images corresponding to the plurality of first sample sub-videos, respectively;

[0064] The first sample image with the earliest timestamp is labeled to obtain labeling information.

[0065] In an optional embodiment, the sample data includes the second sample data, and the second sample data includes: multiple frames of second sample images, second sample audio signals corresponding to each frame of the second sample image, and annotation information corresponding to each frame of the second sample image; the annotation information is used to indicate first position information of pixel points of multiple sounding target objects in each frame of the second sample image;

[0066] The obtaining of sample data includes:

[0067] Acquire a second original sample video and a second original sample audio signal corresponding to the second original sample video, and divide the second original sample video into a plurality of second sample sub-videos;

[0068] Dividing the second original sample audio signal into second sample audio signals corresponding to a plurality of second sample sub-videos, respectively, according to a division method corresponding to the second sample sub-video, and performing frame extraction processing on the plurality of sample sub-videos to obtain second sample images corresponding to the plurality of second sample sub-videos, respectively;

[0069] The plurality of frames of the second sample image are respectively annotated to obtain the annotation information corresponding to the plurality of frames of the second sample image.

[0070] In an optional implementation, the neural network to be trained includes: an encoder to be trained, and a decoder to be trained;

[0071] The method of training the neural network to be trained using the sample data to obtain the target neural network includes:

[0072] Using the encoder to be trained to encode the sample data to obtain sample image feature data and sample audio feature data, and performing feature fusion processing on the sample image feature data and the sample audio feature data to obtain sample fusion feature data;

[0073] Decoding the sample fusion feature data using the decoder to be trained to obtain second position information of a pixel point of a sound object corresponding to the sample audio data in the sample data in the sample video image;

[0074] Determining a model loss based on the sample fusion feature data, the second position information, and the sample audio feature data;

[0075] Parameters of the encoder to be trained and the decoder to be trained are adjusted based on the model loss to obtain a target neural network.

[0076] In a second aspect, an embodiment of the present disclosure further provides a video processing device, including:

[0077] A determination module, configured to determine a target video frame image in a video to be processed and a target audio signal corresponding to the target video frame image;

[0078] an encoding module configured to perform a first encoding process on the target video frame image to obtain image feature data; and perform a second encoding process on the target audio signal to obtain audio feature data;

[0079] A fusion module, configured to perform feature fusion processing on the image feature data and the audio feature data to obtain fused feature data;

[0080] A decoding module is used to perform decoding processing based on the fused feature data to obtain first position information of a pixel point of a sound object corresponding to the audio signal in the target video frame image.

[0081] In an optional implementation manner, the determining module is specifically configured to:

[0082] Acquire the video to be processed and the audio signal to be processed corresponding to the video to be processed, and divide the video to be processed into a plurality of sub-videos to be processed;

[0083] According to the division method corresponding to the sub-video to be processed, the audio signal to be processed is divided into target audio signals corresponding to multiple sub-videos to be processed, and the multiple sub-videos to be processed are respectively subjected to frame extraction processing to obtain target video frame images corresponding to multiple sub-videos to be processed.

[0084] In an optional implementation manner, the encoding module is specifically configured to:

[0085] Performing multi-level feature extraction processing on the target video frame image to obtain intermediate image feature data corresponding to the multi-level feature extraction processing;

[0086] For each level of feature extraction in the multi-level feature extraction, the intermediate image feature data corresponding to each level of feature extraction processing is subjected to parallel sampling of multiple sampling rates of hole convolution to obtain the image feature data corresponding to the target video frame image.

[0087] In an optional implementation, the encoding module is further configured to:

[0088] Converting the target audio signal into frequency spectrum data;

[0089] Perform feature extraction processing on the frequency spectrum data to obtain the audio feature data.

[0090] In an optional implementation, the fusion module includes:

[0091] a conversion unit, configured to perform size conversion processing on the audio feature data to obtain converted audio feature data whose size matches the image feature data;

[0092] a determining unit, configured to determine a similarity between the converted audio feature data and the image feature data;

[0093] An adjustment unit is used to adjust the image feature data based on the similarity to obtain the fused feature data.

[0094] In an optional embodiment, the conversion unit is specifically used for:

[0095] performing linear transformation processing on the audio feature data based on the number of data channels of the image feature data to obtain intermediate audio feature data whose number of data channels is consistent with that of the image feature data;

[0096] Based on the height and width of the image feature data, the intermediate audio feature data is spatially repeated to obtain the converted audio feature data.

[0097] In an optional implementation manner, the determining unit is specifically configured to:

[0098] Performing convolution processing on the image feature data using a first convolution kernel to obtain first convolution image feature data;

[0099] and, performing convolution processing on the converted audio feature data using a second convolution kernel to obtain first convolution-converted audio feature data;

[0100] Performing dimensionality reduction processing on the first convolution image feature data and the first convolution converted audio feature data respectively to obtain second convolution image feature data and second convolution converted audio feature data;

[0101] Performing a dot product process on the second convolution image feature data and the second convolution-converted audio feature data to obtain the similarity.

[0102] In an optional implementation manner, the adjustment unit is specifically configured to:

[0103] Performing convolution processing on the image feature data using a third convolution kernel to obtain third convolution image feature data, and performing dimensionality reduction processing on the third convolution image feature data to obtain fourth convolution image feature data;

[0104] Performing a dot product process on the fourth convolution image feature data and the similarity to obtain adjusted feature data corresponding to the image feature data;

[0105] The adjustment feature data and the image feature data are fused to obtain the fused feature data.

[0106] In an optional implementation, the adjustment unit is further configured to:

[0107] Performing dimensionality upscaling on the adjusted feature data to obtain adjusted feature data after dimensionality upscaling; wherein the data dimension of the adjusted feature data after dimensionality upscaling is the same as the data dimension of the image feature data;

[0108] Using a fourth convolution kernel to perform convolution processing on the dimension-upgraded adjusted feature data to obtain target adjusted feature data;

[0109] The target adjustment feature data and the image feature data are superimposed to obtain the fusion feature data.

[0110] In an optional embodiment, the multi-level feature extraction process includes: a final-level feature extraction process, and a non-final-level feature extraction process; the fused feature data includes: first fused feature data corresponding to the final-level feature extraction process, and second fused feature data corresponding to the non-final-level feature extraction process;

[0111] The decoding module is specifically used for:

[0112] Upsampling the first fused feature data corresponding to the final-level feature extraction process to obtain decoded feature data corresponding to the final-level feature extraction process; and

[0113] fusing the second fused feature data corresponding to the non-final-level feature extraction process and the decoded feature data corresponding to the next-level feature extraction process corresponding to the non-final-level feature extraction to obtain fused feature data corresponding to the non-final-level feature extraction process, and upsampling the fused feature data corresponding to the non-final-level feature extraction process to obtain decoded feature data corresponding to the non-final-level feature extraction process;

[0114] Based on the decoded feature data corresponding to the first-level feature extraction processing in the non-final-level feature extraction processing, the first position information of the pixel point of the sound object corresponding to the audio signal in the target video frame image is obtained.

[0115] In an optional embodiment, the method further includes:

[0116] A generation module is used to generate a mask video frame image including a mask corresponding to the sound object based on first position information of a pixel point of the sound object corresponding to the audio signal in the target video frame image.

[0117] In an optional embodiment, the video processing method is applied to a pre-trained target neural network, wherein the target neural network includes: an encoder network and a decoder network;

[0118] The encoder network is used to perform a first encoding process on the target video frame image to obtain image feature data; and perform a second encoding process on the target audio signal to obtain audio feature data; and perform feature fusion processing on the image feature data and the audio feature data to obtain fused feature data;

[0119] The decoder network is used to perform decoding processing based on the fused feature data to obtain first position information of a pixel point of a sound-emitting object corresponding to the audio signal in the target video frame image.

[0120] In an optional embodiment, the method further includes:

[0121] an acquisition module, configured to acquire sample data; the sample data comprising first sample data for performing semi-supervised training on the neural network, and / or second sample data for performing fully supervised training on the neural network;

[0122] The training module is used to train the neural network to be trained using the sample data to obtain the target neural network.

[0123] In an optional embodiment, the sample data includes the first sample data, and the first sample data includes: multiple frames of first sample images, first sample audio signals corresponding to each frame of the first sample image, and annotation information corresponding to the first frame of the first sample image; the annotation information is used to indicate first position information of a pixel point of a single sounding target object in the first sample image;

[0124] The acquisition module is specifically used for:

[0125] Acquire a first original sample video and a first original sample audio signal corresponding to the first original sample video, and divide the first original sample video into a plurality of first sample sub-videos;

[0126] Dividing the first original sample audio signal into first sample audio signals corresponding to a plurality of first sample sub-videos, respectively, according to a division method corresponding to the first sample sub-video, and performing frame extraction processing on the plurality of first sample sub-videos to obtain first sample images corresponding to the plurality of first sample sub-videos, respectively;

[0127] The first sample image with the earliest timestamp is labeled to obtain labeling information.

[0128] In an optional embodiment, the sample data includes the second sample data, and the second sample data includes: multiple frames of second sample images, second sample audio signals corresponding to each frame of the second sample image, and annotation information corresponding to each frame of the second sample image; the annotation information is used to indicate first position information of pixel points of multiple sounding target objects in each frame of the second sample image;

[0129] The acquisition module is further used for:

[0130] Acquire a second original sample video and a second original sample audio signal corresponding to the second original sample video, and divide the second original sample video into a plurality of second sample sub-videos;

[0131] Dividing the second original sample audio signal into second sample audio signals corresponding to a plurality of second sample sub-videos, respectively, according to a division method corresponding to the second sample sub-video, and performing frame extraction processing on the plurality of sample sub-videos to obtain second sample images corresponding to the plurality of second sample sub-videos, respectively;

[0132] The plurality of frames of the second sample image are respectively annotated to obtain the annotation information corresponding to the plurality of frames of the second sample image.

[0133] In an optional implementation, the neural network to be trained includes: an encoder to be trained, and a decoder to be trained;

[0134] The training module is specifically used for:

[0135] Using the encoder to be trained to encode the sample data to obtain sample image feature data and sample audio feature data, and performing feature fusion processing on the sample image feature data and the sample audio feature data to obtain sample fusion feature data;

[0136] Decoding the sample fusion feature data using the decoder to be trained to obtain second position information of a pixel point of a sound object corresponding to the sample audio data in the sample data in the sample video image;

[0137] Determining a model loss based on the sample fusion feature data, the second position information, and the sample audio feature data;

[0138] Parameters of the encoder to be trained and the decoder to be trained are adjusted based on the model loss to obtain a target neural network.

[0139] In a third aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are performed.

[0140] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0141] For a description of the effects of the above-mentioned video processing device, computer equipment, and computer-readable storage medium, please refer to the description of the above-mentioned video processing method, which will not be repeated here.

[0142] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0143] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0144] Figure 1 A flow chart of a video processing method provided by an embodiment of the present disclosure is shown;

[0145] Figure 2 A schematic diagram of the video processing network structure provided by an embodiment of the present disclosure is shown;

[0146] Figure 3 A schematic diagram showing a fusion process in the video processing method provided by an embodiment of the present disclosure is shown;

[0147] Figure 4 A flowchart of another video processing method provided by an embodiment of the present disclosure is shown;

[0148] Figure 5 A schematic diagram of a video processing device provided by an embodiment of the present disclosure is shown;

[0149] Figure 6 A specific schematic diagram of a fusion module in the video processing device provided by an embodiment of the present disclosure is shown;

[0150] Figure 7 One of the schematic diagrams of another video processing device provided by an embodiment of the present disclosure is shown;

[0151] Figure 8 A second schematic diagram of another video processing device provided by an embodiment of the present disclosure is shown;

[0152] Figure 9 A schematic diagram of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0153] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0154] Research has found that human senses can not only identify objects based on their visual appearance, but also classify them based on the sounds they produce. For example, when hearing a dog barking or a police siren, we can tell that the sound comes from a dog or an ambulance, respectively. This confirms that audio and visual information complement each other. To date, researchers have studied the problem of sound source localization using simplified audio-visual scenarios. Some researchers have studied Advanced Video Coding (AVC) methods to study the correspondence between audio and visual signals in a video. The goal is to determine whether the received audio signal and visual image describe the same scene and perform subsequent processing. That is, based on the phenomenon that audio signals and visual images often occur simultaneously, the corresponding audio-visual scene is studied. Other researchers have also studied sound source localization methods that can classify audio signals and video images and analyze the corresponding audio-visual scene based on the classified audio signals and video images.

[0155] Most of the aforementioned sound source localization methods focus on the problem of sound source localization, that is, locating the sound-making object in a video through audio and video signals. However, they can only achieve rough positioning of large areas, and the positioning accuracy of the sound-making object is low. As a result, the application scenarios corresponding to these methods are limited to the processing level of each group of frames or a short period of time, and the problems that can be solved are limited.

[0156] Based on the above research, the present disclosure provides a video processing method, which obtains the target video frame image and target audio signal in the video to be processed, and encodes the target video frame image and target audio signal to obtain corresponding image feature data and audio feature data, and performs feature fusion processing on the image feature data and audio feature data to obtain fused feature data, and decodes the fused feature data to determine the first position information of the pixel point of the sound object corresponding to the audio signal in the target video frame image. The first position information is the pixel-level position of the sound target in the image, thereby improving the positioning accuracy of the sound object in the video to be processed.

[0157] The defects in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the process of discovering the above problems and the solutions proposed by the present disclosure for the above problems below should be the contributions made by the inventors to the present disclosure during the disclosure process.

[0158] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0159] To facilitate understanding of this embodiment, a video processing method disclosed in this embodiment is first described in detail. The video processing method provided in this embodiment is generally executed by a computer device with certain computing capabilities. In some possible implementations, the video processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0160] The video processing method provided by the embodiment of the present disclosure is described below.

[0161] See also Figure 1 FIG. 1 is a flowchart of a video processing method provided by an embodiment of the present disclosure, wherein the method includes steps S101 to S104, wherein:

[0162] S101: Determine a target video frame image in a video to be processed and a target audio signal corresponding to the target video frame image.

[0163] The video to be processed includes one or more sound-emitting objects, which may be people, animals, objects, etc. When there are multiple sound-emitting objects in the video to be processed, the multiple sound-emitting objects may or may not overlap. The target video frame image is at least one video frame image selected from the video to be processed. The target audio signal is an audio signal captured from the video to be processed and corresponding to the target video frame image.

[0164] In a possible embodiment, a target video frame image in a video to be processed and a target audio signal corresponding to the target video frame image are determined by the following method:

[0165] Acquire the video to be processed and the audio signal to be processed corresponding to the video to be processed, and divide the video to be processed into a plurality of sub-videos to be processed;

[0166] According to the division method corresponding to the sub-video to be processed, the audio signal to be processed is divided into target audio signals corresponding to multiple sub-videos to be processed, and the multiple sub-videos to be processed are respectively subjected to frame extraction processing to obtain target video frame images corresponding to multiple sub-videos to be processed.

[0167] Specifically, the specific method of dividing the video to be processed and dividing the multiple target audio signals and the multiple target video frame images can be determined according to the video processing granularity required by the current business.

[0168] For example, based on the granularity of video processing required by the current business, for example, for a video to be processed that requires relatively fine processing, it can be determined that the video to be processed is divided once per second to obtain multiple sub-videos with a duration of one second. Each sub-video is then subjected to frame extraction processing. That is, according to a pre-set video frame image extraction position, the target video frame image corresponding to each sub-video to be processed is extracted from the multiple video frames of each sub-video to be processed. For example, the i-th frame of each sub-video is extracted as the target video frame image, and the target audio signal corresponding to each sub-video is obtained at the same time. The value of i ranges from 1 to N, where N is the total number of video frames in the sub-video.

[0169] In the embodiment of the present disclosure, when the video processing granularity required by the service is higher, a shorter division time interval may be set; and when the video processing granularity required by the service is lower, a longer division time interval may be set.

[0170] In another possible embodiment, multiple sub-videos to be processed may be divided at intervals of several frames, which is similar to the above method of dividing at fixed time intervals and will not be described in detail here.

[0171] Following S101 above, after determining the target video frame image in the video to be processed and the target audio signal corresponding to the target video frame image, the method further includes:

[0172] S102: Perform a first encoding process on the target video frame image to obtain image feature data; and perform a second encoding process on the target audio signal to obtain audio feature data.

[0173] Specifically, based on the characteristic attributes of the image signal of the target video frame image, it is encoded to obtain its corresponding image feature data, and based on the characteristic attributes of the audio signal of the target audio signal, it is encoded to obtain its corresponding audio feature data.

[0174] The first encoding process is a video encoding process (visual encoder) for extracting video frame features of a target video frame image, and the second encoding process is an audio encoding process (audio encoder) for extracting audio features of a target audio signal.

[0175] In the embodiment of the present disclosure, the target video frame image is subjected to a first encoding process by the following method to obtain image feature data:

[0176] Performing multi-level feature extraction processing on the target video frame image to obtain intermediate image feature data corresponding to the multi-level feature extraction processing;

[0177] For each level of feature extraction in the multi-level feature extraction, the intermediate image feature data corresponding to each level of feature extraction processing is subjected to parallel sampling of multiple sampling rates of hole convolution to obtain the image feature data corresponding to the target video frame image.

[0178] Reference Figure 2 FIG. 1 is a schematic diagram of a video processing network structure provided by an embodiment of the present disclosure. Figure 2 The process shown here, for example, performs four levels of feature extraction on the target video frame, all at the same sampling rate. The feature size of each level is gradually reduced during encoding. This allows the feature map size to be gradually reduced during encoding, reducing the amount of computation. Furthermore, because this is a multi-scale encoding process, the neural network's receptive field progresses from fine-grained to coarse-grained, facilitating the recognition of sound objects of varying sizes.

[0179] Exemplarily, in the disclosed embodiment, an atrous spatial convolutional pyramid pooling (ASPP) network can be used to perform atrous convolution parallel sampling of multiple sampling rates on the intermediate image feature data corresponding to each level of feature extraction processing.

[0180] Specifically, for each level of feature extraction, based on the same sampling rate, the image feature data corresponding to the input of each level is sampled in parallel with the hole convolution to obtain the image feature data of each level of output corresponding to the target video frame image.

[0181] For example, Figure 2As shown, for the first-level feature extraction, the target video frame image is used as input data, and the first-level feature extraction is performed, and the output feature size is The feature data V1, then the feature size is The feature data V1 is input to the next level feature extraction, and the output feature size is The feature data V2 is obtained, and then the feature extraction is performed step by step according to the above method until the feature data of the last level is extracted, and the feature data corresponding to each level is output.

[0182] Among them, T, H, and W represent the characteristic size corresponding to each data channel respectively; C is a constant.

[0183] In the embodiment of the present disclosure, the target audio signal is subjected to a second encoding process by the following method to obtain audio feature data:

[0184] Converting the target audio signal into frequency spectrum data;

[0185] Perform feature extraction processing on the frequency spectrum data to obtain the audio feature data.

[0186] For example, the target audio signal can be converted into spectrum data using a signal processing method such as Fourier transform or short-time Fourier transform. Then, the obtained spectrum data is encoded to obtain the audio feature data A.

[0187] For example, the target audio signal may be subjected to noise reduction or other processing before being converted into spectrum data, thereby reducing errors that may occur in subsequent processing.

[0188] In another possible embodiment, the visual encoder in the encoding process of the target video feature data may also adopt a convolutional neural network, or an attention mechanism neural network (Transformer), etc., which will not be repeated here.

[0189] Following S102 above, after obtaining the image feature data and the audio feature data, the method further includes:

[0190] S103: Perform feature fusion processing on the image feature data and the audio feature data to obtain fused feature data.

[0191] The fused feature data is feature data obtained by processing the image feature data based on the correlation between the audio feature data and the image feature data.

[0192] In the embodiment of the present disclosure, feature fusion processing is performed on the image feature data and the audio feature data through the following steps 1 to 13 to obtain fused feature data:

[0193] Step 11: performing size conversion processing on the audio feature data to obtain converted audio feature data whose size matches the image feature data;

[0194] Step 12: Determine the similarity between the converted audio feature data and the image feature data;

[0195] Step 13: Based on the similarity, adjust the image feature data to obtain the fused feature data.

[0196] Reference Figure 3 As shown, Figure 3 A schematic diagram of the fusion process in the video processing method provided in an embodiment of the present disclosure.

[0197] In step 11, the audio feature data is size-converted to obtain converted audio feature data whose size matches the image feature data, including the following:

[0198] performing linear transformation processing on the audio feature data based on the number of data channels of the image feature data to obtain intermediate audio feature data whose number of data channels is consistent with that of the image feature data;

[0199] Based on the height and width of the image feature data, the intermediate audio feature data is spatially repeated to obtain the converted audio feature data.

[0200] For example, Figure 3 As an example of the fusion process shown in the figure, the feature size of the current image feature data is T×h i ×w i ×C, the corresponding number of data channels is 3, so the audio feature data is transformed into an intermediate audio feature data with a feature size of T×C after the feature size of the audio feature data A is transformed from T×d through the linear layer. Then, based on the height h of the image feature data i , and width w i , repeat the spatial processing of the intermediate audio feature data to obtain a feature size of T×h i ×w i ×C transformed audio feature data.

[0201] Among them, T, H, and W represent the characteristic size corresponding to each data channel respectively; C is a constant.

[0202] In step 1 and step 2, determining the similarity between the converted audio feature data and the image feature data includes the following:

[0203] Performing convolution processing on the image feature data using a first convolution kernel to obtain first convolution image feature data;

[0204] and, performing convolution processing on the converted audio feature data using a second convolution kernel to obtain first convolution-converted audio feature data;

[0205] Performing dimensionality reduction processing on the first convolution image feature data and the first convolution converted audio feature data respectively to obtain second convolution image feature data and second convolution converted audio feature data;

[0206] Performing a dot product process on the second convolution image feature data and the second convolution-converted audio feature data to obtain the similarity.

[0207] For example, Figure 2 As shown, the first convolution kernel θ: 1×1×1 is used to convolve the image feature data to obtain a feature size of T×h i ×w i ×C first convolution image feature data, at the same time, use the second convolution kernel φ: 1×1×1 to convolve the image feature data to obtain a feature size of T×h i ×w i ×C first convolution transformed audio feature data. Afterwards, the first convolution image feature data and the first convolution transformed audio feature data are subjected to dimensionality reduction processing to obtain a feature size of Th i w i ×C second convolution image feature data and feature size is C×Th i w i Finally, the second convolution image feature data and the second convolution audio feature data are multiplied to obtain a feature size of Th. i w i ×Th i w i Similarity is used to represent the correlation information between the converted audio feature data and the image feature data.

[0208] Step 13: Based on the similarity, the image feature data is adjusted to obtain the fused feature data, including the following:

[0209] Performing convolution processing on the image feature data using a third convolution kernel to obtain third convolution image feature data, and performing dimensionality reduction processing on the third convolution image feature data to obtain fourth convolution image feature data;

[0210] Performing a dot product process on the fourth convolution image feature data and the similarity to obtain adjusted feature data corresponding to the image feature data;

[0211] The adjustment feature data and the image feature data are fused to obtain the fused feature data.

[0212] Specifically, the adjustment feature data and the image feature data are fused through the following steps to obtain the fused feature data:

[0213] Performing dimensionality upscaling on the adjusted feature data to obtain adjusted feature data after dimensionality upscaling; wherein the data dimension of the adjusted feature data after dimensionality upscaling is the same as the data dimension of the image feature data;

[0214] Using a fourth convolution kernel to perform convolution processing on the dimension-upgraded adjusted feature data to obtain target adjusted feature data;

[0215] The target adjustment feature data and the image feature data are superimposed to obtain the fusion feature data.

[0216] Exemplarily, after obtaining the similarity between the converted audio feature data and the image feature data, the image feature data is convolved using a third convolution kernel g: 1×1×1 to obtain a feature size of T×h i ×w i The third convolution image feature data of ×C is processed by dimensionality reduction to obtain a feature size of Th i w i After that, the fourth convolution image feature data and the similarity are multiplied to obtain the feature size Th corresponding to the image feature data. i w i After obtaining the adjusted feature data, the adjusted feature data is subjected to dimension-upgrading processing based on the data dimension of the image feature data to obtain the adjusted feature data after dimension-upgrading, whose feature size is T×h i ×w i ×C, and then use the fourth convolution kernel 1×1×1 to perform convolution processing on the adjusted feature data after dimensionality increase to obtain target adjusted feature data. Finally, the target adjusted feature data and the image feature data are superimposed, that is, the fusion process for the image feature data is completed after matrix multiplication of the image feature data and the target adjusted feature data to obtain the fused feature data Z i , whose characteristic size is T×h i ×w i ×C.

[0217] In the embodiment of the present disclosure, regarding the process of feature fusion processing:

[0218] (1): First, the audio feature data is transformed into an intermediate audio feature data with a feature size of T×C after the feature size of the audio feature data A is transformed from T×d through a linear layer. i , and width w i , repeat the spatial processing of the intermediate audio feature data to obtain a feature size of T×h i ×w i ×C transformed audio feature data.

[0219] (2): Secondly, the first convolution kernel θ: 1×1×1 is used to convolve the image feature data to obtain a feature size of T×h i ×w i ×C first convolution image feature data, at the same time, use the second convolution kernel φ: 1×1×1 to convolve the image feature data to obtain a feature size of T×h i ×w i The first convolution transformed audio feature data of ×C is processed by dimensionality reduction respectively on the first convolution image feature data and the first convolution transformed audio feature data, and the feature size is Th i w i ×C second convolution image feature data and feature size is C×Th i w i The second convolution converted audio feature data is processed, and the second convolution image feature data and the second convolution converted audio feature data are processed by point multiplication to obtain a feature size of Th i w i ×Th i w i Finally, the third convolution kernel g: 1×1×1 is used to convolve the image feature data to obtain a feature size of T×h i ×w i The third convolution image feature data of ×C is processed by dimensionality reduction to obtain a feature size of Th i w i The fourth convolution image feature data of ×C.

[0220] (3): Again, after performing dot multiplication on the fourth convolution image feature data and the similarity, the feature size corresponding to the image feature data is obtained as Th i w i After obtaining the adjusted feature data, the adjusted feature data is subjected to dimension-upgrading processing based on the data dimension of the image feature data to obtain the adjusted feature data after dimension-upgrading, whose feature size is T×hi ×w i ×C, and then use the fourth convolution kernel 1×1×1 to perform convolution processing on the adjusted feature data after dimensionality increase to obtain target adjusted feature data,

[0221] (4): Finally, the target adjustment feature data and the image feature data are superimposed, that is, the fusion process for the image feature data is completed by performing matrix multiplication on the image feature data and the target adjustment feature data to obtain the fused feature data Z i , whose characteristic size is T×h i ×w i ×C. By performing feature fusion processing on the image feature data and the audio feature data to obtain fused feature data, two modal information of the target video frame image data and the target audio data can be encoded simultaneously, solving the related problems of multimodal signals.

[0222] Following S103 above, after obtaining the fused feature data, the method includes:

[0223] Decoding is performed based on the fused feature data to obtain first position information of a pixel point of a sound-emitting object corresponding to the audio signal in the target video frame image.

[0224] The first position information of the pixel point of the sound object corresponding to the audio signal in the target video frame image is the position information of multiple pixel points corresponding to the sound object in the target video frame image.

[0225] Specifically, according to the description of steps S102 to S103 in the embodiment of the present disclosure, it can be seen that the multi-level feature extraction processing includes: final-level feature extraction processing, and non-final-level feature extraction processing; the fused feature data includes: first fused feature data corresponding to the final-level feature extraction processing, and second fused feature data corresponding to the non-final-level feature extraction processing.

[0226] In the embodiment of the present disclosure, based on the above description information, decoding processing can be performed based on the fused feature data through the following steps to obtain first position information of the sound object corresponding to the audio signal in the target video frame image:

[0227] Upsampling the first fused feature data corresponding to the final-level feature extraction process to obtain decoded feature data corresponding to the final-level feature extraction process; and

[0228] fusing the second fused feature data corresponding to the non-final-level feature extraction process and the decoded feature data corresponding to the next-level feature extraction process corresponding to the non-final-level feature extraction to obtain fused feature data corresponding to the non-final-level feature extraction process, and upsampling the fused feature data corresponding to the non-final-level feature extraction process to obtain decoded feature data corresponding to the non-final-level feature extraction process;

[0229] Based on the decoded feature data corresponding to the first-level feature extraction processing in the non-final-level feature extraction processing, the first position information of the pixel point of the sound object corresponding to the audio signal in the target video frame image is obtained.

[0230] For example, Figure 2 As an example of the video processing process shown in FIG, for the final decoding process, the first fusion feature data is upsampled to obtain a feature size of The decoded feature data P1 is then fed into the upsampling network at the previous level. After upsampling at each level, the first position information of the pixel of the sound-emitting object corresponding to the audio signal in the target video frame image is output as mask M. The corresponding feature size is the same as that of the target video frame image. This process, through multiple cascaded stages of reverse decoding, can output decoded feature data P with gradually increasing feature sizes corresponding to each decoding stage.

[0231] The disclosed embodiment is based on the pixel-level audio-visual segmentation (AVS) technology, which uses an encoder-decoder network structure and can directly output a sound source localization mask that is consistent with the pixel size of the input image. Specifically, for the encoding process, multi-level feature extraction can be performed. For the first-level feature extraction, the target video frame image is used as input data. After the first-level feature extraction, the output feature size is The feature data V1, then the feature size is The feature data V1 is input to the next level feature extraction, and the output feature size is The feature data V2 is obtained, and then the feature extraction is performed step by step according to the above method until the feature data of the last level is extracted, and the video feature data corresponding to each level is output. At the same time, the target audio signal is subjected to a second encoding process to obtain audio feature data. After that, the image feature data and the audio feature data are subjected to feature fusion processing to obtain fused feature data. Finally, the fused feature data is subjected to a multi-level decoding process. For the final level decoding process, the first fused feature data is upsampled to obtain a feature size of The decoded feature data P1 is then input into the previous upsampling network. After upsampling step by step, the first position information of the pixel point of the sound object corresponding to the audio signal in the target video frame image is output, which is the mask M, and the corresponding feature size is the same as the feature size of the target video frame image.

[0232] In another possible embodiment of the present disclosure, the present invention further includes:

[0233] Based on first position information of a pixel point of a sound-emitting object corresponding to the audio signal in the target video frame image, a mask video frame image including a mask corresponding to the sound-emitting object is generated.

[0234] The masked video frame image is a video frame image including a mask M, wherein the mask M corresponds to one or more sound-emitting objects. Figure 2 In the example, the masked video frame image is the final output image.

[0235] In another embodiment of the present disclosure, the video processing method is applied to a pre-trained target neural network, wherein the target neural network includes: an encoder network and a decoder network;

[0236] The encoder network is used to perform a first encoding process on the target video frame image to obtain image feature data; and perform a second encoding process on the target audio signal to obtain audio feature data; and perform feature fusion processing on the image feature data and the audio feature data to obtain fused feature data;

[0237] The decoder network is used to perform decoding processing based on the fused feature data to obtain first position information of a pixel point of a sound-emitting object corresponding to the audio signal in the target video frame image.

[0238] The specific operation of the encoder network and the decoder network is as shown in the above steps S101 to S104, which will not be repeated here.

[0239] See also Figure 4 FIG. 4 is a flowchart of a target neural network training method according to an embodiment of the present disclosure. The method includes steps S401 to S402, wherein:

[0240] S401: Acquire sample data; the sample data includes first sample data for performing semi-supervised training on the neural network, and / or second sample data for performing fully supervised training on the neural network.

[0241] The sample data includes first sample data and second sample data. Full supervision and semi-supervision have different corresponding known label information. Full supervision adds corresponding annotation information to sample images of all frames, while semi-supervision only has the annotation information of the first frame. Semi-supervision is aimed at relatively simple single-source problems, while full supervision is aimed at relatively complex multi-source problems.

[0242] When the sample data includes the first sample data, i.e., the first sample data for semi-supervised training of a neural network, the first sample data includes: multiple frames of first sample images, first sample audio signals corresponding to each frame of the first sample image, and annotation information corresponding to the first frame of the first sample image; the annotation information is used to indicate the first position information of the pixel point of the target object of a single sound in the first sample image. In this case, obtaining the sample data includes: obtaining a first original sample video and a first original sample audio signal corresponding to the first original sample video, and dividing the first original sample video into multiple first sample sub-videos; dividing the first original sample audio signal into first sample audio signals corresponding to multiple first sample sub-videos according to the division method corresponding to the first sample sub-video, and performing frame extraction processing on the multiple first sample sub-videos to obtain first sample images corresponding to the multiple first sample sub-videos; and annotating the first sample image with the earliest timestamp to obtain annotation information.

[0243] Exemplarily, when the sample image includes only a single sounding object, a first original sample video is obtained and the corresponding first original sample audio signal is extracted. Based on business needs, the first original sample video is divided into multiple first sample sub-videos, for example, a first sample sub-video is divided every second, and the first sample audio signal corresponding to each first sample sub-video is extracted. A first sample image frame is extracted for each first sample sub-video, and annotation information is added to the first sample image to represent the first position information of the pixel point of the single sounding target object in the first sample image, so as to facilitate subsequent semi-supervised neural network training based on the first sample data.

[0244] When the sample data includes the second sample data, i.e., the second sample data for fully supervised training of the neural network, the second sample data includes: multiple frames of second sample images, second sample audio signals corresponding to each frame of the second sample image, and annotation information corresponding to each frame of the second sample image; the annotation information is used to indicate the first position information of the pixel points of multiple sounding target objects in each frame of the second sample image. In this case, obtaining the sample data includes: obtaining a second original sample video and a second original sample audio signal corresponding to the second original sample video, and dividing the second original sample video into multiple second sample sub-videos; dividing the second original sample audio signal into second sample audio signals corresponding to multiple second sample sub-videos according to the division method corresponding to the second sample sub-videos; and performing frame extraction processing on the multiple sample sub-videos to obtain second sample images corresponding to the multiple second sample sub-videos; and annotating the multiple frames of the second sample images to obtain annotation information corresponding to the multiple frames of the second sample images.

[0245] Exemplarily, when the sample image includes multiple sound-emitting objects, a second original sample video is obtained and the corresponding second original sample audio signal is extracted. Based on business needs, the second original sample video is divided into multiple second sample sub-videos, for example, a second sample sub-video is divided every second, and the second sample audio signal corresponding to each second sample sub-video is extracted. A second sample image frame is extracted for each second sample sub-video, and annotation information is added to the second sample image to represent the second position information of the pixel point of each sound-emitting target object in the second sample image, so as to facilitate subsequent fully supervised neural network training based on the second sample data.

[0246] In an embodiment of the present disclosure, the first sample data and the second sample data both include a target video frame image obtained from the video to be processed without encoding-decoding processing, and a mask video frame image that has been encoded-decoded and includes first position information of pixel points of the sound-emitting object corresponding to the audio signal in the target video frame image.

[0247] Following the above step S401, after obtaining the sample data, the following steps are further included:

[0248] S402: Using the sample data to train the neural network to be trained, to obtain the target neural network.

[0249] The neural network to be trained includes an encoder to be trained and a decoder to be trained.

[0250] Specifically, the target neural network is obtained by training the neural network to be trained using the sample data through the following content:

[0251] Using the encoder to be trained to encode the sample data to obtain sample image feature data and sample audio feature data, and performing feature fusion processing on the sample image feature data and the sample audio feature data to obtain sample fusion feature data;

[0252] Decoding the sample fusion feature data using the decoder to be trained to obtain second position information of a pixel point of a sound object corresponding to the sample audio data in the sample data in the sample video image;

[0253] Determining a model loss based on the sample fusion feature data, the second position information, and the sample audio feature data;

[0254] Parameters of the encoder to be trained and the decoder to be trained are adjusted based on the model loss to obtain a target neural network.

[0255] In a possible embodiment of the present disclosure, a method for determining module loss includes: in the process of training the model, based on the output data of the model each time and the real sample data, calculating the model loss value corresponding to each set of data, and continuously adjusting the model parameters based on the model loss value until the model loss value corresponding to the output data of the model and the real sample data is less than a preset loss threshold, completing the model training process and obtaining the target neural network.

[0256] In another possible embodiment of the present disclosure, the method for determining model loss also includes: obtaining mask image feature data of the mask video frame image, performing dot product processing on the mask image feature data and the fusion feature data, and performing average pooling processing on the feature data after the dot product processing to obtain average image feature data; and performing linear feature transformation on the target audio signal to obtain linear audio feature data, determining divergence data of the average image feature data and the linear audio feature data, and verifying the feature matching degree between the first position information of the sound-emitting object in the target video frame image and the target audio signal based on the divergence data.

[0257] For example, in order to constrain the masked video frame image output by the model to segment the real sound-making object in the video to be processed, an audio-visual matching loss function L_AVM is designed to constrain the semantic matching degree of the image feature data and audio feature data in the masked video frame image. The specific calculation method is:

[0258]

[0259] Among them, KL represents the relative entropy (Kullback–Leibler, KL) divergence, M i The final output mask video frame image M of the neural network is obtained by downsampling, which has the same size as Zi, where Z i is the fusion feature data; ⊙ represents the dot product, avg represents the average pooling, A i It is obtained by performing a linear transformation on the audio feature data A.

[0260] In the embodiment of the present disclosure, KL divergence is used to measure the similarity between image feature data and audio feature data, and Euclidean distance or the like may also be used.

[0261] In another embodiment of the present disclosure, if the audio features of certain video frame images are relatively close in the feature space, then the corresponding sound objects should be relatively close in the feature space. In this embodiment of the present disclosure, the calculation of the total objective function L is set as follows:

[0262] L=BCE(M,Y)+λL AVM (M,Z,A)

[0263] Where λ is a balancing weight, ⊙ represents element-wise multiplication, BCE is the binary cross entropy loss, and Y is the pixel-level annotation. For semi-supervised training, the audiovisual regularization loss is meaningless, so λ can be set to 0.

[0264] Based on the above loss function, it is possible to constrain whether the masked video frame image has segmented the real sound-making object in the video to be processed, thereby improving the accuracy of the neural network output and making the generated masked video frame image more accurately represent the position information corresponding to the sound.

[0265] The disclosed embodiment obtains a target video frame image and a target audio signal from a video to be processed, encodes the target video frame image and the target audio signal to obtain corresponding image feature data and audio feature data, performs feature fusion processing on the image feature data and the audio feature data to obtain fused feature data, and decodes the fused feature data to determine the first position information of the pixel point of the sound-emitting object in the target video frame image corresponding to the audio signal. In this way, the accuracy of locating the sound-emitting object in the video to be processed can be improved.

[0266] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0267] Based on the same inventive concept, the embodiment of the present disclosure also provides a video processing device corresponding to the video processing method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned video processing method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0268] Reference Figure 5 FIG. 1 is a schematic diagram of a video processing device provided by an embodiment of the present disclosure, wherein the device includes: a determination module 510, an encoding module 520, a fusion module 530, and a decoding module 540; wherein,

[0269] A determination module 510 is configured to determine a target video frame image in a video to be processed and a target audio signal corresponding to the target video frame image;

[0270] The encoding module 520 is configured to perform a first encoding process on the target video frame image to obtain image feature data; and perform a second encoding process on the target audio signal to obtain audio feature data;

[0271] A fusion module 530 is configured to perform feature fusion processing on the image feature data and the audio feature data to obtain fused feature data;

[0272] The decoding module 540 is configured to perform decoding processing based on the fused feature data to obtain first position information of a pixel point of a sound object corresponding to the audio signal in the target video frame image.

[0273] In an optional implementation, the determining module 510 is specifically configured to:

[0274] Acquire the video to be processed and the audio signal to be processed corresponding to the video to be processed, and divide the video to be processed into a plurality of sub-videos to be processed;

[0275] According to the division method corresponding to the sub-video to be processed, the audio signal to be processed is divided into target audio signals corresponding to multiple sub-videos to be processed, and the multiple sub-videos to be processed are respectively subjected to frame extraction processing to obtain target video frame images corresponding to multiple sub-videos to be processed.

[0276] In an optional implementation manner, the encoding module 520 is specifically configured to:

[0277] Performing multi-level feature extraction processing on the target video frame image to obtain intermediate image feature data corresponding to the multi-level feature extraction processing;

[0278] For each level of feature extraction in the multi-level feature extraction, the intermediate image feature data corresponding to each level of feature extraction processing is subjected to parallel sampling of multiple sampling rates of hole convolution to obtain the image feature data corresponding to the target video frame image.

[0279] In an optional implementation, the encoding module 520 is further configured to:

[0280] Converting the target audio signal into frequency spectrum data;

[0281] Perform feature extraction processing on the frequency spectrum data to obtain the audio feature data.

[0282] In an optional embodiment, referring to Figure 6 FIG. 5 is a schematic diagram of a fusion module in a video processing device according to an embodiment of the present disclosure; the fusion module 530 includes:

[0283] a conversion unit 531 configured to perform size conversion processing on the audio feature data to obtain converted audio feature data having a size matching that of the image feature data;

[0284] a determining unit 532, configured to determine a similarity between the converted audio feature data and the image feature data;

[0285] The adjustment unit 533 is configured to adjust the image feature data based on the similarity to obtain the fused feature data.

[0286] In an optional embodiment, the conversion unit 531 is specifically configured to:

[0287] performing linear transformation processing on the audio feature data based on the number of data channels of the image feature data to obtain intermediate audio feature data whose number of data channels is consistent with that of the image feature data;

[0288] Based on the height and width of the image feature data, the intermediate audio feature data is spatially repeated to obtain the converted audio feature data.

[0289] In an optional implementation, the determining unit 532 is specifically configured to:

[0290] Performing convolution processing on the image feature data using a first convolution kernel to obtain first convolution image feature data;

[0291] and, performing convolution processing on the converted audio feature data using a second convolution kernel to obtain first convolution-converted audio feature data;

[0292] Performing dimensionality reduction processing on the first convolution image feature data and the first convolution converted audio feature data respectively to obtain second convolution image feature data and second convolution converted audio feature data;

[0293] Performing a dot product process on the second convolution image feature data and the second convolution-converted audio feature data to obtain the similarity.

[0294] In an optional implementation, the adjusting unit 533 is specifically configured to:

[0295] Performing convolution processing on the image feature data using a third convolution kernel to obtain third convolution image feature data, and performing dimensionality reduction processing on the third convolution image feature data to obtain fourth convolution image feature data;

[0296] Performing a dot product process on the fourth convolution image feature data and the similarity to obtain adjusted feature data corresponding to the image feature data;

[0297] The adjustment feature data and the image feature data are fused to obtain the fused feature data.

[0298] In an optional implementation, the adjusting unit 533 is further configured to:

[0299] Performing dimensionality upscaling on the adjusted feature data to obtain adjusted feature data after dimensionality upscaling; wherein the data dimension of the adjusted feature data after dimensionality upscaling is the same as the data dimension of the image feature data;

[0300] Using a fourth convolution kernel to perform convolution processing on the dimension-upgraded adjusted feature data to obtain target adjusted feature data;

[0301] The target adjustment feature data and the image feature data are superimposed to obtain the fusion feature data.

[0302] In an optional embodiment, the multi-level feature extraction process includes: a final-level feature extraction process, and a non-final-level feature extraction process; the fused feature data includes: first fused feature data corresponding to the final-level feature extraction process, and second fused feature data corresponding to the non-final-level feature extraction process;

[0303] The decoding module 540 is specifically configured to:

[0304] Upsampling the first fused feature data corresponding to the final-level feature extraction process to obtain decoded feature data corresponding to the final-level feature extraction process; and

[0305] fusing the second fused feature data corresponding to the non-final-level feature extraction process and the decoded feature data corresponding to the next-level feature extraction process corresponding to the non-final-level feature extraction to obtain fused feature data corresponding to the non-final-level feature extraction process, and upsampling the fused feature data corresponding to the non-final-level feature extraction process to obtain decoded feature data corresponding to the non-final-level feature extraction process;

[0306] Based on the decoded feature data corresponding to the first-level feature extraction processing in the non-final-level feature extraction processing, the first position information of the pixel point of the sound object corresponding to the audio signal in the target video frame image is obtained.

[0307] In an optional embodiment, referring to Figure 6 FIG. 1 is a schematic diagram of another video processing device provided by an embodiment of the present disclosure; and further includes:

[0308] The generating module 550 is configured to generate a mask video frame image including a mask corresponding to the sound-emitting object based on first position information of a pixel point of the sound-emitting object corresponding to the audio signal in the target video frame image.

[0309] In an optional embodiment, the video processing method is applied to a pre-trained target neural network, wherein the target neural network includes: an encoder network and a decoder network;

[0310] The encoder network is used to perform a first encoding process on the target video frame image to obtain image feature data; and perform a second encoding process on the target audio signal to obtain audio feature data; and perform feature fusion processing on the image feature data and the audio feature data to obtain fused feature data;

[0311] The decoder network is used to perform decoding processing based on the fused feature data to obtain first position information of a pixel point of a sound-emitting object corresponding to the audio signal in the target video frame image.

[0312] In an optional embodiment, referring to Figure 8 FIG. 2 is a second schematic diagram of another video processing device provided by an embodiment of the present disclosure; the device further includes:

[0313] An acquisition module 810 is configured to acquire sample data; the sample data includes first sample data for performing semi-supervised training on the neural network and / or second sample data for performing fully supervised training on the neural network;

[0314] The training module 820 is used to train the neural network to be trained using the sample data to obtain the target neural network.

[0315] In an optional embodiment, the sample data includes the first sample data, and the first sample data includes: multiple frames of first sample images, first sample audio signals corresponding to each frame of the first sample image, and annotation information corresponding to the first frame of the first sample image; the annotation information is used to indicate first position information of a pixel point of a single sounding target object in the first sample image;

[0316] The acquisition module 810 is specifically used for:

[0317] Acquire a first original sample video and a first original sample audio signal corresponding to the first original sample video, and divide the first original sample video into a plurality of first sample sub-videos;

[0318] Dividing the first original sample audio signal into first sample audio signals corresponding to a plurality of first sample sub-videos, respectively, according to a division method corresponding to the first sample sub-video, and performing frame extraction processing on the plurality of first sample sub-videos to obtain first sample images corresponding to the plurality of first sample sub-videos, respectively;

[0319] The first sample image with the earliest timestamp is labeled to obtain labeling information.

[0320] In an optional embodiment, the sample data includes the second sample data, and the second sample data includes: multiple frames of second sample images, second sample audio signals corresponding to each frame of the second sample image, and annotation information corresponding to each frame of the second sample image; the annotation information is used to indicate first position information of pixel points of multiple sounding target objects in each frame of the second sample image;

[0321] The acquisition module 810 is further configured to:

[0322] Acquire a second original sample video and a second original sample audio signal corresponding to the second original sample video, and divide the second original sample video into a plurality of second sample sub-videos;

[0323] Dividing the second original sample audio signal into second sample audio signals corresponding to a plurality of second sample sub-videos, respectively, according to a division method corresponding to the second sample sub-video, and performing frame extraction processing on the plurality of sample sub-videos to obtain second sample images corresponding to the plurality of second sample sub-videos, respectively;

[0324] The plurality of frames of the second sample image are respectively annotated to obtain the annotation information corresponding to the plurality of frames of the second sample image.

[0325] In an optional implementation, the neural network to be trained includes: an encoder to be trained, and a decoder to be trained;

[0326] The training module 820 is specifically used to:

[0327] Using the encoder to be trained to encode the sample data to obtain sample image feature data and sample audio feature data, and performing feature fusion processing on the sample image feature data and the sample audio feature data to obtain sample fusion feature data;

[0328] Decoding the sample fusion feature data using the decoder to be trained to obtain second position information of a pixel point of a sound object corresponding to the sample audio data in the sample data in the sample video image;

[0329] Determining a model loss based on the sample fusion feature data, the second position information, and the sample audio feature data;

[0330] Parameters of the encoder to be trained and the decoder to be trained are adjusted based on the model loss to obtain a target neural network.

[0331] The disclosed embodiment obtains the target video frame image and target audio signal in the video to be processed, encodes the target video frame image and target audio signal, obtains the corresponding image feature data and audio feature data, performs feature fusion processing on the image feature data and audio feature data, obtains fused feature data, decodes the fused feature data, and determines the first position information of the pixel point of the sound object corresponding to the audio signal in the target video frame image. In this way, the positioning accuracy of the sound object in the video to be processed can be improved.

[0332] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0333] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 9900 , which includes a processor 901 , a memory 902 , and a bus 903 . The memory 902 is used to store execution instructions and includes a memory 9021 and an external memory 9022 . The memory 9021 , also referred to as internal memory, is used to temporarily store computational data in the processor 901 and data exchanged with an external memory 9022 , such as a hard disk. The processor 901 exchanges data with the external memory 9022 via the memory 9021 . When the computer device 900 is running, the processor 901 communicates with the memory 902 via the bus 903 , enabling the processor 901 to execute the following instructions:

[0334] Determining a target video frame image in a video to be processed and a target audio signal corresponding to the target video frame image;

[0335] Performing a first encoding process on the target video frame image to obtain image feature data; and performing a second encoding process on the target audio signal to obtain audio feature data;

[0336] Performing feature fusion processing on the image feature data and the audio feature data to obtain fused feature data;

[0337] Decoding is performed based on the fused feature data to obtain first position information of a pixel point of a sound-emitting object corresponding to the audio signal in the target video frame image.

[0338] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes the steps of the video processing method described in the above method embodiment. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0339] The computer program product of the video processing method provided in the embodiments of the present disclosure includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the video processing method described in the above method embodiments. For details, please refer to the above method embodiments and will not be repeated here.

[0340] The present disclosure also provides a computer program that, when executed by a processor, implements any of the methods of the aforementioned embodiments. The computer program product can be implemented in hardware, software, or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0341] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0342] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0343] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0344] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0345] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A video processing method, characterized in that: include: Determining a target video frame image in a video to be processed and a target audio signal corresponding to the target video frame image; Performing a first encoding process on the target video frame image to obtain image feature data; and performing a second encoding process on the target audio signal to obtain audio feature data; Performing feature fusion processing on the image feature data and the audio feature data to obtain fused feature data; Decoding is performed based on the fused feature data to obtain first position information of a pixel point of a sound object corresponding to the audio signal in the target video frame image; wherein, The performing feature fusion processing on the image feature data and the audio feature data to obtain fused feature data includes: Performing size conversion processing on the audio feature data to obtain converted audio feature data whose size matches the image feature data; Determining a similarity between the converted audio feature data and the image feature data; Based on the similarity, the image feature data is adjusted to obtain the fused feature data; wherein, Determining the similarity between the converted audio feature data and the image feature data includes: Performing convolution processing on the image feature data using a first convolution kernel to obtain first convolution image feature data; and, performing convolution processing on the converted audio feature data using a second convolution kernel to obtain first convolution-converted audio feature data; Performing dimensionality reduction processing on the first convolution image feature data and the first convolution-converted audio feature data respectively to obtain second convolution image feature data and second convolution-converted audio feature data; Performing a dot product process on the second convolution image feature data and the second convolution-converted audio feature data to obtain the similarity; The adjusting and processing the image feature data based on the similarity to obtain the fused feature data includes: Performing convolution processing on the image feature data using a third convolution kernel to obtain third convolution image feature data, and performing dimensionality reduction processing on the third convolution image feature data to obtain fourth convolution image feature data; Performing a dot product process on the fourth convolution image feature data and the similarity to obtain adjusted feature data corresponding to the image feature data; The adjustment feature data and the image feature data are fused to obtain the fused feature data; wherein, The fusing the adjustment feature data and the image feature data to obtain the fused feature data includes: Performing dimensionality upscaling on the adjusted feature data to obtain adjusted feature data after dimensionality upscaling; wherein the data dimension of the adjusted feature data after dimensionality upscaling is the same as the data dimension of the image feature data; Using a fourth convolution kernel to perform convolution processing on the dimension-upgraded adjusted feature data to obtain target adjusted feature data; The target adjustment feature data and the image feature data are superimposed to obtain the fusion feature data.

2. The video processing method according to claim 1, wherein: The determining of a target video frame image in a video to be processed and a target audio signal corresponding to the target video frame image includes: Acquire the video to be processed and the audio signal to be processed corresponding to the video to be processed, and divide the video to be processed into a plurality of sub-videos to be processed; According to the division method corresponding to the sub-video to be processed, the audio signal to be processed is divided into target audio signals corresponding to multiple sub-videos to be processed, and the multiple sub-videos to be processed are respectively subjected to frame extraction processing to obtain target video frame images corresponding to multiple sub-videos to be processed.

3. The video processing method according to claim 1, wherein: The performing a first encoding process on the target video frame image to obtain image feature data includes: Performing multi-level feature extraction processing on the target video frame image to obtain intermediate image feature data corresponding to the multi-level feature extraction processing; For each level of feature extraction in the multi-level feature extraction, the intermediate image feature data corresponding to each level of feature extraction processing is subjected to parallel sampling of multiple sampling rates of hole convolution to obtain the image feature data corresponding to the target video frame image.

4. The video processing method according to claim 1, wherein: Performing a second encoding process on the target audio signal to obtain audio feature data includes: Converting the target audio signal into frequency spectrum data; Perform feature extraction processing on the frequency spectrum data to obtain the audio feature data.

5. The video processing method according to claim 1, wherein: The step of performing size conversion processing on the audio feature data to obtain converted audio feature data having a size matching the image feature data includes: performing linear transformation processing on the audio feature data based on the number of data channels of the image feature data to obtain intermediate audio feature data whose number of data channels is consistent with that of the image feature data; Based on the height and width of the image feature data, the intermediate audio feature data is spatially repeated to obtain the converted audio feature data.

6. The video processing method according to claim 3, wherein: The multi-level feature extraction process includes: a final-level feature extraction process and a non-final-level feature extraction process; the fused feature data includes: first fused feature data corresponding to the final-level feature extraction process and second fused feature data corresponding to the non-final-level feature extraction process; The decoding process based on the fused feature data to obtain first position information of the sound object corresponding to the audio signal in the target video frame image includes: Upsampling the first fused feature data corresponding to the final-level feature extraction process to obtain decoded feature data corresponding to the final-level feature extraction process; and fusing the second fused feature data corresponding to the non-final-level feature extraction process and the decoded feature data corresponding to the next-level feature extraction process corresponding to the non-final-level feature extraction to obtain fused feature data corresponding to the non-final-level feature extraction process, and upsampling the fused feature data corresponding to the non-final-level feature extraction process to obtain decoded feature data corresponding to the non-final-level feature extraction process; Based on the decoded feature data corresponding to the first-level feature extraction processing in the non-final-level feature extraction processing, the first position information of the pixel point of the sound object corresponding to the audio signal in the target video frame image is obtained.

7. The video processing method according to any one of claims 1 to 6, characterized in that: Also includes: Based on first position information of a pixel point of a sound-emitting object corresponding to the audio signal in the target video frame image, a mask video frame image including a mask corresponding to the sound-emitting object is generated.

8. The video processing method according to any one of claims 1 to 6, characterized in that: The video processing method is applied to a pre-trained target neural network, wherein the target neural network includes an encoder network and a decoder network; The encoder network is used to perform a first encoding process on the target video frame image to obtain image feature data; and perform a second encoding process on the target audio signal to obtain audio feature data; and perform feature fusion processing on the image feature data and the audio feature data to obtain fused feature data; The decoder network is used to perform decoding processing based on the fused feature data to obtain first position information of a pixel point of a sound-emitting object corresponding to the audio signal in the target video frame image.

9. The video processing method according to claim 8, characterized in that: Also includes: Acquire sample data; the sample data includes first sample data for performing semi-supervised training on the neural network, and / or second sample data for performing fully supervised training on the neural network; The neural network to be trained is trained using the sample data to obtain the target neural network.

10. The video processing method according to claim 9, characterized in that: The sample data includes the first sample data, which includes: a plurality of frames of first sample images, first sample audio signals corresponding to each frame of the first sample image, and annotation information corresponding to the first frame of the first sample image; the annotation information is used to indicate first position information of a pixel point of a single sounding target object in the first sample image; The obtaining of sample data includes: Acquire a first original sample video and a first original sample audio signal corresponding to the first original sample video, and divide the first original sample video into a plurality of first sample sub-videos; Dividing the first original sample audio signal into first sample audio signals corresponding to a plurality of first sample sub-videos, respectively, according to a division method corresponding to the first sample sub-video, and performing frame extraction processing on the plurality of first sample sub-videos to obtain first sample images corresponding to the plurality of first sample sub-videos, respectively; The first sample image with the earliest timestamp is labeled to obtain labeling information.

11. The video processing method according to claim 9 or 10, characterized in that: The sample data includes the second sample data, the second sample data including: a plurality of frames of second sample images, a second sample audio signal corresponding to each frame of the second sample image, and annotation information corresponding to each frame of the second sample image; the annotation information is used to indicate first position information of pixel points of a plurality of sounding target objects in each frame of the second sample image; The obtaining of sample data includes: Acquire a second original sample video and a second original sample audio signal corresponding to the second original sample video, and divide the second original sample video into a plurality of second sample sub-videos; Dividing the second original sample audio signal into second sample audio signals corresponding to a plurality of second sample sub-videos, respectively, according to a division method corresponding to the second sample sub-video, and performing frame extraction processing on the plurality of sample sub-videos to obtain second sample images corresponding to the plurality of second sample sub-videos, respectively; The plurality of frames of the second sample image are respectively annotated to obtain the annotation information corresponding to the plurality of frames of the second sample image.

12. The video processing method according to claim 9 or 10, characterized in that: The neural network to be trained includes: an encoder to be trained and a decoder to be trained; The method of training the neural network to be trained using the sample data to obtain the target neural network includes: Using the encoder to be trained to encode the sample data to obtain sample image feature data and sample audio feature data, and performing feature fusion processing on the sample image feature data and the sample audio feature data to obtain sample fusion feature data; Decoding the sample fusion feature data using the decoder to be trained to obtain second position information of a pixel point of a sound object corresponding to the sample audio data in the sample data in the sample video image; Determining a model loss based on the sample fusion feature data, the second position information, and the sample audio feature data; Parameters of the encoder to be trained and the decoder to be trained are adjusted based on the model loss to obtain a target neural network.

13. A video processing device, characterized in that: include: A determination module, configured to determine a target video frame image in a video to be processed and a target audio signal corresponding to the target video frame image; an encoding module configured to perform a first encoding process on the target video frame image to obtain image feature data; and perform a second encoding process on the target audio signal to obtain audio feature data; A fusion module, configured to perform feature fusion processing on the image feature data and the audio feature data to obtain fused feature data; A decoding module is used to perform decoding processing based on the fusion feature data to obtain first position information of a pixel point of a sound object corresponding to the audio signal in the target video frame image; wherein, The performing feature fusion processing on the image feature data and the audio feature data to obtain fused feature data includes: Performing size conversion processing on the audio feature data to obtain converted audio feature data whose size matches the image feature data; Determining a similarity between the converted audio feature data and the image feature data; Based on the similarity, the image feature data is adjusted to obtain the fused feature data; wherein, Determining the similarity between the converted audio feature data and the image feature data includes: Performing convolution processing on the image feature data using a first convolution kernel to obtain first convolution image feature data; and, performing convolution processing on the converted audio feature data using a second convolution kernel to obtain first convolution-converted audio feature data; Performing dimensionality reduction processing on the first convolution image feature data and the first convolution-converted audio feature data respectively to obtain second convolution image feature data and second convolution-converted audio feature data; Performing a dot product process on the second convolution image feature data and the second convolution-converted audio feature data to obtain the similarity; The adjusting and processing the image feature data based on the similarity to obtain the fused feature data includes: Performing convolution processing on the image feature data using a third convolution kernel to obtain third convolution image feature data, and performing dimensionality reduction processing on the third convolution image feature data to obtain fourth convolution image feature data; Performing a dot product process on the fourth convolution image feature data and the similarity to obtain adjusted feature data corresponding to the image feature data; The adjustment feature data and the image feature data are fused to obtain the fused feature data; wherein, The fusing the adjustment feature data and the image feature data to obtain the fused feature data includes: Performing dimensionality upscaling on the adjusted feature data to obtain adjusted feature data after dimensionality upscaling; wherein the data dimension of the adjusted feature data after dimensionality upscaling is the same as the data dimension of the image feature data; Using a fourth convolution kernel to perform convolution processing on the dimension-upgraded adjusted feature data to obtain target adjusted feature data; The target adjustment feature data and the image feature data are superimposed to obtain the fusion feature data.

14. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the video processing method according to any one of claims 1 to 12 are performed.

15. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the video processing method according to any one of claims 1 to 12.

Images