Audio reconstruction method and device for image-assisted audio completion

Abstract

The present invention discloses an audio reconstruction method and device for image-assisted audio completion, the method comprising: performing a preprocessing operation on a video to be processed according to a video preprocessing scheme to obtain audio and image data, wherein the audio data is data obtained after at least one of blank audio removal, sampling rate conversion and feature extraction operations; performing a primary reconstruction operation on the audio data according to an audio reconstruction model to obtain a vector to be fused; performing a secondary reconstruction operation on the dimensional vector of the fused vector and the image data to obtain a secondary reconstruction result, the secondary reconstruction result is used to determine an audio loss parameter of the audio reconstruction model, and when it is determined that the audio loss parameter indicates that the model convergence parameter of the audio reconstruction model is within a preset convergence threshold, it is determined that the training task for the audio reconstruction model is completed. It can be seen that the implementation of the present invention can realize audio reconstruction based on images, improve the accuracy of audio reconstruction and improve the matching degree between the reconstructed audio and the video content.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to an audio reconstruction method and device for image-assisted audio completion. Background Art

[0002] With the popularity of smartphones, smartphones have become an indispensable entertainment tool for people. Among the many options for people to use smartphones for entertainment, watching short videos is one of the most popular choices. For users who are gradually accustomed to the short-term stimulation of short videos, in addition to the main content of short videos, people's considerations for the number of views and the length of video viewing are more reflected in the music and background sounds of short videos. The reason is that choosing a vocal music that matches the content of short videos is more likely to drive users' emotions and realize the value of short videos.

[0003] However, in the vast amount of short videos, most of the music materials have been reused many times. Inevitably, repeated background music will only bring aesthetic fatigue to users' viewing and listening senses. However, the creative ideas of short video creators are not endless. How to find music that matches the emotions of the video content / story has become a creative problem for many short video creators. It can be seen that it is particularly important to provide a method to solve the mismatch between video soundtracks and video content / story emotions. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an audio reconstruction method and device for image-assisted audio completion, which can realize audio reconstruction based on image-assisted audio, improve the accuracy and reliability of audio reconstruction, and also help to improve the matching degree between the reconstructed audio and the video content.

[0005] In order to solve the above technical problems, the first aspect of the present invention discloses an audio reconstruction method for image-assisted audio completion, the method comprising:

[0006] Performing a preprocessing operation on the acquired video to be processed according to a preset video preprocessing scheme to obtain separated data corresponding to the video to be processed, wherein the separated data includes audio data and image data, and the audio data is data obtained after at least one of a blank audio removal operation, a sampling rate conversion operation, and a feature extraction operation;

[0007] Performing a primary reconstruction operation on the audio data according to a preset audio reconstruction model to obtain a primary reconstruction result corresponding to the audio data, wherein the primary reconstruction result includes a vector to be fused, and the vector to be fused is used to be fused with a dimensional vector corresponding to the image data to reconstruct the video to be processed, and the vector to be fused is consistent with the dimensional vector in feature dimension;

[0008] According to the audio reconstruction model, a secondary reconstruction operation is performed on the vector to be fused and the dimensional vector to obtain a secondary reconstruction result corresponding to the video to be processed, wherein the secondary reconstruction result is used to determine an audio loss parameter of the audio reconstruction model, and the audio loss parameter is used to determine a training progress of the audio reconstruction model;

[0009] When it is determined that the audio loss parameter indicates that the model convergence parameter of the audio reconstruction model is within a preset convergence threshold, it is determined that the training task for the audio reconstruction model is completed, and the trained audio reconstruction model is used to perform audio reconstruction operations on the audio to be reconstructed.

[0010] As an optional implementation, in the first aspect of the present invention, performing a secondary reconstruction operation on the vector to be fused and the dimensional vector according to the audio reconstruction model to obtain a secondary reconstruction result corresponding to the video to be processed includes:

[0011] According to the audio reconstruction model and the preset vector dimension standard, a preset feature vector processing operation is performed on the vector to be fused and the dimension vector to obtain an audio processing vector corresponding to the vector to be fused, wherein the audio processing vector includes an audio identifier for representing global information of the audio processing vector;

[0012] According to the audio reconstruction model and the audio identifier, an audio reconstruction operation is performed on the audio processing vector and the dimensional vector to obtain a secondary reconstruction result corresponding to the video to be processed.

[0013] As an optional implementation, in the first aspect of the present invention, the step of performing a preset feature vector processing operation on the vector to be fused and the dimensional vector according to a preset vector dimension standard according to the audio reconstruction model to obtain an audio processing vector corresponding to the vector to be fused includes:

[0014] According to the audio reconstruction model, performing vector splicing and dimensional transformation operations on the vector to be fused and the dimensional vector at a preset dimensional level to obtain a splicing transformation vector corresponding to the vector to be fused;

[0015] Inputting the splicing transformation vector into a preset fully connected layer to obtain an output vector corresponding to the splicing transformation vector;

[0016] The nonlinear transformation operation, the dimensional transformation operation, the category labeling operation and the mean conversion operation are sequentially performed on the output vector to obtain a mean conversion result corresponding to the output vector as the audio processing vector corresponding to the vector to be fused.

[0017] As an optional implementation, in the first aspect of the present invention, performing an audio reconstruction operation on the audio processing vector and the dimensional vector according to the audio reconstruction model and the audio identifier to obtain a secondary reconstruction result corresponding to the video to be processed includes:

[0018] According to the fully connected layer included in the audio reconstruction model, a feature fusion operation is performed on the audio processing vector to obtain a feature fusion result corresponding to the audio processing vector;

[0019] According to the audio identifier, performing a feature processing operation on the feature fusion result to obtain a feature processing vector corresponding to the feature fusion result;

[0020] An audio loss parameter between the audio processing vector and the feature processing vector is calculated, and the audio loss parameter is used to determine the training progress of the audio reconstruction model.

[0021] As an optional implementation manner, in the first aspect of the present invention, performing a feature processing operation on the feature fusion result according to the audio identifier to obtain a feature processing vector corresponding to the feature fusion result includes:

[0022] Acquire a non-fused vector, where the non-fused vector is other vectors in the first-level reconstruction result except the vector to be fused;

[0023] According to the audio identifier and the non-fused vector, a feature parsing operation is performed on the feature fusion result to obtain a parsing vector corresponding to the feature fusion result;

[0024] The audio identifier in the parsing vector is deleted, and the updated parsing vector is determined as the feature processing vector corresponding to the feature fusion result.

[0025] As an optional implementation, in the first aspect of the present invention, the first-level reconstruction operation is specifically an audio masking operation, and the performing the first-level reconstruction operation on the audio data according to the preset audio reconstruction model to obtain the first-level reconstruction result corresponding to the audio data includes:

[0026] determining a target interval for performing the audio masking operation on the audio data;

[0027] Setting a selection probability corresponding to each frame of data in the audio data;

[0028] According to a preset audio reconstruction model, a selection probability corresponding to each frame of data, and the target interval, the audio mask operation is performed on the audio data to obtain a first-level reconstruction result corresponding to the audio data.

[0029] As an optional implementation, in the first aspect of the present invention, performing a preprocessing operation on the video to be processed according to a preset video preprocessing scheme to obtain separated data corresponding to the video to be processed includes:

[0030] According to a preset video preprocessing scheme, a preprocessing operation is performed on the video to be processed to obtain initial audio data and image data corresponding to the video to be processed;

[0031] Analyze the initial audio data to obtain audio parameters of the initial audio data, where the audio parameters include audio length and audio sampling rate;

[0032] Determine whether the audio parameters of the initial audio data match the standard audio parameters, and when the determination result is yes, perform a convolution processing operation on the initial audio data according to a preset convolution layer to obtain first audio data corresponding to the initial audio data, and determine the first audio data and the image data as separated data corresponding to the video to be processed;

[0033] When the judgment result is no, a parameter conversion operation is performed on the initial audio data, and the obtained parameter conversion result is input into a preset convolution layer to obtain second audio data corresponding to the initial audio data, and the second audio data and the image data are determined as separated data corresponding to the video to be processed.

[0034] A second aspect of the present invention discloses an audio reconstruction device for image-assisted audio completion, the device comprising:

[0035] A preprocessing module, used to perform a preprocessing operation on the acquired video to be processed according to a preset video preprocessing scheme to obtain separated data corresponding to the video to be processed, wherein the separated data includes audio data and image data, and the audio data is data obtained after at least one of a blank audio removal operation, a sampling rate conversion operation, and a feature extraction operation;

[0036] A first reconstruction module is used to perform a first-level reconstruction operation on the audio data according to a preset audio reconstruction model to obtain a first-level reconstruction result corresponding to the audio data, wherein the first-level reconstruction result includes a vector to be fused, and the vector to be fused is used to be fused with a dimensional vector corresponding to the image data to reconstruct the video to be processed, and the vector to be fused is consistent with the dimensional vector in feature dimension;

[0037] A second reconstruction module is used to perform a secondary reconstruction operation on the vector to be fused and the dimensional vector according to the audio reconstruction model to obtain a secondary reconstruction result corresponding to the video to be processed, wherein the secondary reconstruction result is used to determine an audio loss parameter of the audio reconstruction model, and the audio loss parameter is used to determine a training progress of the audio reconstruction model;

[0038] A determination module is used to determine that the training task for the audio reconstruction model is completed when it is determined that the audio loss parameter indicates that the model convergence parameter of the audio reconstruction model is within a preset convergence threshold, and the trained audio reconstruction model is used to perform audio reconstruction operations on the audio to be reconstructed.

[0039] As an optional implementation, in the second aspect of the present invention, the second reconstruction module includes:

[0040] A feature processing submodule, configured to perform a preset feature vector processing operation on the vector to be fused and the dimension vector according to the preset vector dimension standard according to the audio reconstruction model, to obtain an audio processing vector corresponding to the vector to be fused, wherein the audio processing vector includes an audio identifier for representing global information of the audio processing vector;

[0041] The reconstruction submodule is used to perform an audio reconstruction operation on the audio processing vector and the dimensional vector according to the audio reconstruction model and the audio identifier to obtain a secondary reconstruction result corresponding to the video to be processed.

[0042] As an optional implementation, in the second aspect of the present invention, the feature processing submodule performs a preset feature vector processing operation on the vector to be fused and the dimensional vector according to the preset vector dimension standard according to the audio reconstruction model, and the method of obtaining the audio processing vector corresponding to the vector to be fused specifically includes:

[0043] According to the audio reconstruction model, performing vector splicing and dimensional transformation operations on the vector to be fused and the dimensional vector at a preset dimensional level to obtain a splicing transformation vector corresponding to the vector to be fused;

[0044] Inputting the splicing transformation vector into a preset fully connected layer to obtain an output vector corresponding to the splicing transformation vector;

[0045] The nonlinear transformation operation, the dimensional transformation operation, the category labeling operation and the mean conversion operation are sequentially performed on the output vector to obtain a mean conversion result corresponding to the output vector as the audio processing vector corresponding to the vector to be fused.

[0046] As an optional implementation, in the second aspect of the present invention, the reconstruction submodule performs an audio reconstruction operation on the audio processing vector and the dimensional vector according to the audio reconstruction model and the audio identifier, and obtains a secondary reconstruction result corresponding to the video to be processed, specifically including:

[0047] According to the fully connected layer included in the audio reconstruction model, a feature fusion operation is performed on the audio processing vector to obtain a feature fusion result corresponding to the audio processing vector;

[0048] According to the audio identifier, performing a feature processing operation on the feature fusion result to obtain a feature processing vector corresponding to the feature fusion result;

[0049] An audio loss parameter between the audio processing vector and the feature processing vector is calculated, and the audio loss parameter is used to determine the training progress of the audio reconstruction model.

[0050] As an optional implementation, in the second aspect of the present invention, the reconstruction submodule performs a feature processing operation on the feature fusion result according to the audio identifier, and a method of obtaining a feature processing vector corresponding to the feature fusion result specifically includes:

[0051] Acquire a non-fused vector, where the non-fused vector is other vectors in the first-level reconstruction result except the vector to be fused;

[0052] According to the audio identifier and the non-fused vector, a feature parsing operation is performed on the feature fusion result to obtain a parsing vector corresponding to the feature fusion result;

[0053] The audio identifier in the parsing vector is deleted, and the updated parsing vector is determined as the feature processing vector corresponding to the feature fusion result.

[0054] As an optional implementation, in the second aspect of the present invention, the first-level reconstruction operation is specifically an audio masking operation, and the first reconstruction module performs a first-level reconstruction operation on the audio data according to a preset audio reconstruction model, and a method for obtaining a first-level reconstruction result corresponding to the audio data specifically includes:

[0055] determining a target interval for performing the audio masking operation on the audio data;

[0056] Setting a selection probability corresponding to each frame of data in the audio data;

[0057] According to a preset audio reconstruction model, a selection probability corresponding to each frame of data, and the target interval, the audio mask operation is performed on the audio data to obtain a first-level reconstruction result corresponding to the audio data.

[0058] As an optional implementation, in the second aspect of the present invention, the preprocessing module performs a preprocessing operation on the video to be processed according to a preset video preprocessing scheme, and a method of obtaining separated data corresponding to the video to be processed specifically includes:

[0059] According to a preset video preprocessing scheme, a preprocessing operation is performed on the video to be processed to obtain initial audio data and image data corresponding to the video to be processed;

[0060] Analyze the initial audio data to obtain audio parameters of the initial audio data, where the audio parameters include audio length and audio sampling rate;

[0061] Determine whether the audio parameters of the initial audio data match the standard audio parameters, and when the determination result is yes, perform a convolution processing operation on the initial audio data according to a preset convolution layer to obtain first audio data corresponding to the initial audio data, and determine the first audio data and the image data as separated data corresponding to the video to be processed;

[0062] When the judgment result is no, a parameter conversion operation is performed on the initial audio data, and the obtained parameter conversion result is input into a preset convolution layer to obtain second audio data corresponding to the initial audio data, and the second audio data and the image data are determined as separated data corresponding to the video to be processed.

[0063] A third aspect of the present invention discloses another audio reconstruction device for image-assisted audio completion, the device comprising:

[0064] A memory storing executable program code;

[0065] a processor coupled to the memory;

[0066] The processor calls the executable program code stored in the memory to execute the audio reconstruction method for image-assisted audio completion disclosed in the first aspect of the present invention.

[0067] A fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when called, are used to execute the audio reconstruction method for image-assisted audio completion disclosed in the first aspect of the present invention.

[0068] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0069] In an embodiment of the present invention, an audio reconstruction method for image-assisted audio completion is provided, the method comprising: performing a preprocessing operation on an acquired video to be processed according to a preset video preprocessing scheme to obtain separated data corresponding to the video to be processed, the separated data comprising audio data and image data, the audio data being data obtained after at least one of a blank audio removal operation, a sampling rate conversion operation, and a feature extraction operation; performing a primary reconstruction operation on the audio data according to a preset audio reconstruction model to obtain a primary reconstruction result corresponding to the audio data, the primary reconstruction result comprising a vector to be fused, the vector to be fused being used to fuse with a dimensional vector corresponding to the image data to reconstruct the video to be processed, the vector to be fused being consistent with the dimensional vector in feature dimension; performing a secondary reconstruction operation on the vector to be fused and the dimensional vector according to the audio reconstruction model to obtain a secondary reconstruction result corresponding to the video to be processed, the secondary reconstruction result being used to determine an audio loss parameter of the audio reconstruction model, the audio loss parameter being used to determine the training progress of the audio reconstruction model; when it is determined that the audio loss parameter indicates that the model convergence parameter of the video reconstruction model is within a preset convergence threshold, it is determined that the training task for the audio reconstruction model is completed, and the trained audio reconstruction model is used to perform an audio reconstruction operation on the reconstructed audio. It can be seen that the implementation of the present invention can automatically perform preprocessing on the processed video to obtain image data and audio data. The image data and audio data obtained by the pre-processing reduce the amount of data required for subsequent audio reconstruction model training, and improve the training efficiency of the subsequent audio reconstruction model training; then, the audio reconstruction model is intelligently performed a first-level reconstruction operation on the audio data, and a second-level reconstruction operation is performed on the fusion vector and the dimensional vector, thereby finally completing the training of the audio reconstruction model. The finally trained audio reconstruction model is used to perform audio reconstruction operations on the reconstructed audio, thereby improving the accuracy and reliability of the obtained reconstructed audio. In addition, the reconstructed audio improves the richness of the audio material and improves the matching degree between the audio material and the adaptation content corresponding to the audio material, which is beneficial to improving the audience's viewing experience of the reconstructed audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0071] Figure 1 It is a flowchart of an audio reconstruction method for image-assisted audio completion disclosed in an embodiment of the present invention;

[0072] Figure 2It is a flowchart of another audio reconstruction method for image-assisted audio completion disclosed in an embodiment of the present invention;

[0073] Figure 3 It is a structural schematic diagram of an audio reconstruction device for image-assisted audio completion disclosed in an embodiment of the present invention;

[0074] Figure 4 is a structural schematic diagram of another audio reconstruction device for image-assisted audio completion disclosed in an embodiment of the present invention;

[0075] Figure 5 It is a structural schematic diagram of another audio reconstruction device for image-assisted audio completion disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or end including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or ends.

[0078] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0079] The present invention discloses an audio reconstruction method and device for image-assisted audio completion, which can automatically perform preprocessing on a processed video to obtain image data and audio data. The image data and audio data obtained by pre-processing reduce the amount of data required for subsequent audio reconstruction model training, and improve the training efficiency of the subsequent audio reconstruction model; and then the audio reconstruction model intelligently performs a first-level reconstruction operation on the audio data, and performs a second-level reconstruction operation on the fusion vector and the dimension vector, thereby finally completing the training of the audio reconstruction model. The finally trained audio reconstruction model is used to perform an audio reconstruction operation on the reconstructed audio, thereby improving the accuracy and reliability of the obtained reconstructed audio. In addition, the reconstructed audio improves the richness of the audio material and improves the matching degree between the audio material and the adaptation content corresponding to the audio material, which is conducive to improving the audience's viewing experience of the reconstructed audio. The following are detailed descriptions.

[0080] Embodiment 1

[0081] See also Figure 1 , Figure 1 is a flow chart of an audio reconstruction method for image-assisted audio completion disclosed in an embodiment of the present invention. Figure 1 The described audio reconstruction method for image-assisted audio completion can be applied to an audio reconstruction device for image-assisted audio completion, which is not limited in the embodiments of the present invention. Figure 1 As shown, the audio reconstruction method of image-assisted audio completion may include the following operations:

[0082] 101. Perform a preprocessing operation on the acquired video to be processed according to a preset video preprocessing scheme to obtain separated data corresponding to the video to be processed, where the separated data includes audio data and image data.

[0083] In an embodiment of the present invention, the audio data is data obtained after at least one of a blank audio removal operation, a sampling rate conversion operation, and a feature extraction operation; wherein, in order to reduce the interference of redundant and blank audio on subsequent operations and avoid the audio being too long, when it is detected that the length of the audio data is too long, it is necessary to perform a blank audio removal operation; and, in order to facilitate the processing efficiency of subsequent processing of the audio data, the sampling rate of the audio data will be uniformly converted; in addition, finally, feature extraction will be performed on the basis of the audio data by fusion processing of the feature data of the audio data.

[0084] 102. Perform a first-level reconstruction operation on the audio data according to a preset audio reconstruction model to obtain a first-level reconstruction result corresponding to the audio data, where the first-level reconstruction result includes a vector to be fused.

[0085] In the embodiment of the present invention, the vector to be fused is used to be fused with the dimensional vector corresponding to the image data to reconstruct the video to be processed, and the vector to be fused is consistent with the dimensional vector in terms of feature dimension.

[0086] In the embodiment of the present invention, optionally, the primary reconstruction operation is specifically an audio mask operation, and step 102 performs the primary reconstruction operation on the audio data according to the preset audio reconstruction model, and the method of obtaining the primary reconstruction result corresponding to the audio data may specifically include the following operations:

[0087] determining a target interval for performing an audio masking operation on the audio data;

[0088] Setting a selection probability corresponding to each frame of audio data;

[0089] According to the preset audio reconstruction model, the selection probability corresponding to each frame of data and the target interval, an audio masking operation is performed on the audio data to obtain a first-level reconstruction result corresponding to the audio data.

[0090] 103. According to the audio reconstruction model, a secondary reconstruction operation is performed on the fused vector and the dimension vector to obtain a secondary reconstruction result corresponding to the video to be processed.

[0091] In the embodiment of the present invention, the secondary reconstruction result is used to determine the audio loss parameter of the audio reconstruction model, and the audio loss parameter is used to determine the training progress of the audio reconstruction model.

[0092] 104. When it is determined that the audio loss parameter indicates that the model convergence parameter of the audio reconstruction model is within a preset convergence threshold, it is determined that the training task for the audio reconstruction model is completed.

[0093] In the embodiment of the present invention, the trained audio reconstruction model is used to perform an audio reconstruction operation on the audio to be reconstructed.

[0094] In an embodiment of the present invention, when it is determined that the audio loss parameter indicates that the model convergence parameter of the audio reconstruction model is outside a preset convergence threshold, the audio reconstruction model is back-propagated according to the audio loss parameter to update the model parameters adopted by the audio reconstruction model.

[0095] It can be seen that implementation Figure 1The described audio reconstruction method for image-assisted audio completion can automatically perform preprocessing on the processed video to obtain image data and audio data. The image data and audio data obtained by pre-processing reduce the amount of data required for subsequent audio reconstruction model training, and improve the training efficiency of the subsequent audio reconstruction model; then, the audio reconstruction model intelligently performs a first-level reconstruction operation on the audio data, and a second-level reconstruction operation on the fusion vector and the dimensional vector, thereby finally completing the training of the audio reconstruction model. The finally trained audio reconstruction model is used to perform audio reconstruction operations on the reconstructed audio, thereby improving the accuracy and reliability of the obtained reconstructed audio. In addition, the reconstructed audio improves the richness of the audio material and improves the matching degree between the audio material and the adaptation content corresponding to the audio material, which is conducive to improving the audience's viewing experience of the reconstructed audio.

[0096] Embodiment 2

[0097] See also Figure 2 , Figure 2 is a flow chart of another audio reconstruction method for image-assisted audio completion disclosed in an embodiment of the present invention. Figure 2 The described audio reconstruction method for image-assisted audio completion can be applied to an audio reconstruction device for image-assisted audio completion, which is not limited in the embodiments of the present invention. Figure 2 As shown, the audio reconstruction method of image-assisted audio completion may include the following operations:

[0098] 201. According to a preset video preprocessing scheme, a preprocessing operation is performed on the video to be processed to obtain initial audio data and image data corresponding to the video to be processed.

[0099] 202. Analyze the initial audio data to obtain audio parameters of the initial audio data, where the audio parameters include audio length and audio sampling rate.

[0100] 203. Determine whether the audio parameters of the initial audio data match the standard audio parameters.

[0101] When the judgment result of step 203 is yes, step 204 is executed, and when the judgment result of step 203 is no, step 205 is executed.

[0102] 204. Perform a convolution processing operation on the initial audio data according to a preset convolution layer to obtain first audio data corresponding to the initial audio data, and determine the first audio data and the image data as separated data corresponding to the video to be processed.

[0103] 205. Perform a parameter conversion operation on the initial audio data, and input the obtained parameter conversion result into a preset convolution layer to obtain second audio data corresponding to the initial audio data, and determine the second audio data and the image data as separated data corresponding to the video to be processed.

[0104] 206. Perform a first-level reconstruction operation on the audio data according to a preset audio reconstruction model to obtain a first-level reconstruction result corresponding to the audio data, where the first-level reconstruction result includes a vector to be fused.

[0105] 207. According to the audio reconstruction model, perform a secondary reconstruction operation on the fused vector and the dimension vector to obtain a secondary reconstruction result corresponding to the video to be processed.

[0106] 208. When it is determined that the audio loss parameter indicates that the model convergence parameter of the audio reconstruction model is within a preset convergence threshold, it is determined that the training task for the audio reconstruction model is completed.

[0107] In the embodiment of the present invention, for other descriptions of step 206 to step 208, please refer to other specific descriptions of step 102 to step 104 in the first embodiment, and the embodiment of the present invention will not be repeated here.

[0108] It can be seen that implementation Figure 2 The described audio reconstruction method for image-assisted audio completion can automatically analyze the audio parameters of the initial audio data obtained based on the separation of the video to be processed, and then compare them with the standard audio parameters. For the audio parameters that do not match the standard audio parameters, the parameters can be automatically converted, and finally the audio parameters are processed through a convolutional layer to obtain the required audio data, thereby reducing the amount of parameters and calculations required for subsequent audio reconstruction model training, reducing the probability of model training errors, and to a certain extent improving the subsequent audio reconstruction model training success rate, as well as improving the reconstruction success rate of the audio reconstruction model for the audio to be reconstructed and the reliability of the reconstructed audio.

[0109] In an optional embodiment, step 207 performs a secondary reconstruction operation on the fusion vector and the dimension vector according to the audio reconstruction model, and the method of obtaining the secondary reconstruction result corresponding to the video to be processed may specifically include the following operations:

[0110] According to the audio reconstruction model and the preset vector dimension standard, a preset feature vector processing operation is performed on the vector to be fused and the dimension vector to obtain an audio processing vector corresponding to the vector to be fused, wherein the audio processing vector includes an audio identifier for representing global information of the audio processing vector;

[0111] According to the audio reconstruction model and the audio identifier, an audio reconstruction operation is performed on the audio processing vector and the dimension vector to obtain a secondary reconstruction result corresponding to the video to be processed.

[0112] In this optional embodiment, optionally, the above-mentioned method of performing a preset feature vector processing operation on the to-be-fused vector and the dimensional vector according to the preset vector dimension standard according to the audio reconstruction model to obtain the audio processing vector corresponding to the to-be-fused vector may include the following operations:

[0113] According to the audio reconstruction model, vector splicing and dimension transformation operations are performed on the to-be-fused vector and the dimension vector at a preset dimension level to obtain a splicing transformation vector corresponding to the to-be-fused vector;

[0114] Input the splicing transformation vector into a preset fully connected layer to obtain an output vector corresponding to the splicing transformation vector;

[0115] The nonlinear transformation operation, dimensionality transformation operation, category labeling operation and mean conversion operation are sequentially performed on the output vector to obtain the mean conversion result corresponding to the output vector as the audio processing vector corresponding to the vector to be fused.

[0116] In this optional embodiment, optionally, the above-mentioned method of performing the audio reconstruction operation on the audio processing vector and the dimensional vector according to the audio reconstruction model and the audio identifier to obtain the secondary reconstruction result corresponding to the video to be processed may include the following operations:

[0117] According to the fully connected layer included in the audio reconstruction model, a feature fusion operation is performed on the audio processing vector to obtain a feature fusion result corresponding to the audio processing vector;

[0118] According to the audio identifier, a feature processing operation is performed on the feature fusion result to obtain a feature processing vector corresponding to the feature fusion result;

[0119] An audio loss parameter is calculated between the audio processing vector and the feature processing vector, and the audio loss parameter is used to determine the training progress of the audio reconstruction model.

[0120] Furthermore, the above method of performing a feature processing operation on the feature fusion result according to the audio identifier to obtain a feature processing vector corresponding to the feature fusion result may include the following operations:

[0121] Obtaining non-fused vectors, where the non-fused vectors are other vectors in the first-level reconstruction result except the vector to be fused;

[0122] According to the audio identifier and the non-fused vector, a feature parsing operation is performed on the feature fusion result to obtain a parsing vector corresponding to the feature fusion result;

[0123] The audio identifier in the parsing vector is deleted, and the updated parsing vector is determined as the feature processing vector corresponding to the feature fusion result.

[0124] It can be seen that in this optional embodiment, the vector to be processed and the dimensional vector can be automatically fused according to the vector dimension standard according to the audio reconstruction model, thereby improving the accuracy of the reconstruction and fusion of audio data and image data; further, the audio processing vector obtained based on the pre-fusion operation is combined with the audio identifier and the dimensional vector, and input into the audio reconstruction model to perform reconstruction training of the audio reconstruction model, thereby improving the training efficiency of the audio reconstruction model and the accuracy and reliability of the audio reconstruction model; in addition, the trained and accurate and reliable audio reconstruction model is used to reconstruct the audio to be reconstructed, thereby improving the accuracy of the reconstructed audio, enriching the material content of the reconstructed audio, and also improving the matching degree between the reconstructed audio and the image content corresponding to the reconstructed audio, thereby improving the viewing experience of the audience of the reconstructed audio (including images) to a certain extent.

[0125] In another optional embodiment, the above-mentioned manner of performing convolution processing on the initial audio data or the parameter conversion result may specifically include the following operations:

[0126] Determine the target number of convolutional layers corresponding to the convolutional layer used to process the input data;

[0127] According to the target number of convolution layers and the preset convolution processing scheme, a convolution operation is performed on the input data to obtain a convolution operation result corresponding to the input data, where the convolution operation result includes a convolution dimension value of the input data;

[0128] Determine whether the convolution dimension value of the input data is greater than the preset dimension value. If the judgment result is yes, perform a data cleaning operation on the convolution operation results that are greater than the preset dimension value to update the convolution operation results. The convolution dimension value corresponding to the updated convolution operation result is equal to the preset dimension value.

[0129] When it is determined that the convolution dimension value of the input data is less than the preset dimension value, the convolution operation result is vector-complemented according to a preset dimension completion scheme so that the convolution dimension value in the convolution operation result is equal to the preset dimension value.

[0130] In this optional embodiment, optionally, after determining that the convolution dimension value in the convolution operation result is equal to the preset dimension value, the method may further include the following operations:

[0131] Determine the target clustering value, which is used to perform clustering operation on the convolution operation results;

[0132] Performing a clustering operation on the convolution operation result to obtain a clustering result corresponding to the convolution operation result, wherein the clustering result includes a plurality of audio feature vectors corresponding to the convolution operation result, the value of the audio feature vector is the target cluster value, and each audio feature vector corresponds to a plurality of sub-feature vectors;

[0133] According to all audio feature vectors, a target encoding table is constructed, and the target encoding table is used to convert a continuous audio vector into a discrete audio vector;

[0134] For each audio vector in the convolution operation result, calculate the similarity parameter between each audio vector and each audio feature vector in the target coding table, obtain the target parameter with the smallest parameter value corresponding to the similarity parameter between the audio vector and each audio feature vector in the target coding table, and the target audio vector corresponding to the target parameter, where the target audio vector is an audio feature vector in the target coding table;

[0135] According to the target audio vector, the audio vector in the convolution operation result is updated to update the continuous audio vector in the convolution operation result to a discrete audio vector. The updated convolution operation result is used to determine the loss function, which is used to determine the model convergence value of the subsequent audio reconstruction model.

[0136] For this optional embodiment, an example is given as follows:

[0137] A convolution layer of N1 layers is pre-constructed, the input audio length is N2 seconds, and the sampling rate is converted to a specific FskHz, then the speech signal can be expressed as (N2, Fs); assuming that the predetermined audio length is L1, the conversion vector obtained after passing through the convolution layer of the N1 layer can be expressed as (N2, L1, W1), W1 is the dimension of the input speech after conversion, and when the dimension of W1 is less than the preset dimension value, it is completed, and the updated conversion vector is expressed as (N2, L1, W2), where W2 is the dimension of the conversion vector, which is equal to W1+L1; wherein, the audio is input to each convolution layer, and the audio is continuously subjected to two-dimensional convolution and two-dimensional maximum pooling layers in the frequency domain and time domain to reduce the feature dimension of the input audio. The role of the convolution layer is to help learn the time domain information of the audio signal on the one hand; on the other hand, it can compress the time slice dimension of the audio signal to save calculation. The values ​​of the above-mentioned parameters can be adjusted according to the actual application, and the embodiments of the present invention are not limited thereto.

[0138] It can be seen that in this optional embodiment, data processing can be automatically performed on the input data through the preset convolutional layer and clustering scheme to convert the input data into the required audio vector, thereby reducing the amount of data required to be processed by the subsequent audio reconstruction model, which is beneficial to improving the model training efficiency and model training accuracy of the audio reconstruction model.

[0139] Embodiment 3

[0140] See also Figure 3 , Figure 3 It is a structural diagram of an audio reconstruction device for image-assisted audio completion disclosed in an embodiment of the present invention. Among them, the audio reconstruction device for image-assisted audio completion can be an audio reconstruction terminal for image-assisted audio completion, an audio reconstruction device for image-assisted audio completion, an audio reconstruction system for image-assisted audio completion, or an audio reconstruction server for image-assisted audio completion. The audio reconstruction server for image-assisted audio completion can be a local server, a remote server, or a cloud server (also called a cloud server). When the audio reconstruction server for image-assisted audio completion is a non-cloud server, the non-cloud server can communicate with the cloud server, which is not limited in the embodiment of the present invention. Figure 3 As shown, the audio reconstruction device for image-assisted audio completion may include a preprocessing module 301, a first reconstruction module 302, a second reconstruction module 303 and a determination module 304, wherein:

[0141] The preprocessing module 301 is used to perform preprocessing operations on the acquired video to be processed according to a preset video preprocessing scheme to obtain separated data corresponding to the video to be processed. The separated data includes audio data and image data. The audio data is data obtained after at least one of blank audio removal operation, sampling rate conversion operation and feature extraction operation.

[0142] The first reconstruction module 302 is used to perform a first-level reconstruction operation on the audio data according to a preset audio reconstruction model to obtain a first-level reconstruction result corresponding to the audio data. The first-level reconstruction result includes a vector to be fused. The vector to be fused is used to be fused with a dimensional vector corresponding to the image data to reconstruct the video to be processed. The vector to be fused is consistent with the dimensional vector in terms of feature dimension.

[0143] The second reconstruction module 303 is used to perform a secondary reconstruction operation on the vector to be fused and the dimensional vector obtained by the first reconstruction module 302 according to the audio reconstruction model, so as to obtain a secondary reconstruction result corresponding to the video to be processed. The secondary reconstruction result is used to determine the audio loss parameters of the audio reconstruction model, and the audio loss parameters are used to determine the training progress of the audio reconstruction model.

[0144] The determination module 304 is used to determine that the training task for the audio reconstruction model is completed when it is determined that the audio loss parameter represents that the model convergence parameter of the audio reconstruction model is within a preset convergence threshold, and the trained audio reconstruction model is used to perform audio reconstruction operations on the audio to be reconstructed.

[0145] In the embodiment of the present invention, optionally, the primary reconstruction operation is specifically an audio mask operation, and the first reconstruction module 302 performs the primary reconstruction operation on the audio data according to the preset audio reconstruction model, and the manner in which the primary reconstruction result corresponding to the audio data is obtained specifically includes:

[0146] determining a target interval for performing an audio masking operation on the audio data;

[0147] Setting a selection probability corresponding to each frame of audio data;

[0148] According to the preset audio reconstruction model, the selection probability corresponding to each frame of data and the target interval, an audio masking operation is performed on the audio data to obtain a first-level reconstruction result corresponding to the audio data.

[0149] In the embodiment of the present invention, optionally, the preprocessing module 301 performs a preprocessing operation on the video to be processed according to a preset video preprocessing scheme, and obtains the separation data corresponding to the video to be processed in a manner specifically including:

[0150] According to a preset video preprocessing scheme, a preprocessing operation is performed on the video to be processed to obtain initial audio data and image data corresponding to the video to be processed;

[0151] Analyze the initial audio data to obtain audio parameters of the initial audio data, where the audio parameters include audio length and audio sampling rate;

[0152] Determine whether the audio parameters of the initial audio data match the standard audio parameters. If the determination result is yes, perform a convolution processing operation on the initial audio data according to a preset convolution layer to obtain first audio data corresponding to the initial audio data, and determine the first audio data and the image data as separated data corresponding to the video to be processed;

[0153] When the judgment result is no, a parameter conversion operation is performed on the initial audio data, and the obtained parameter conversion result is input into a preset convolution layer to obtain second audio data corresponding to the initial audio data, and the second audio data and image data are determined as separated data corresponding to the video to be processed.

[0154] It can be seen that implementation Figure 3The described audio reconstruction device for image-assisted audio completion can automatically perform preprocessing on the processed video to obtain image data and audio data. The image data and audio data obtained by pre-processing reduce the amount of data required for subsequent audio reconstruction model training, thereby improving the training efficiency of the subsequent audio reconstruction model training; and then the audio reconstruction model intelligently performs a first-level reconstruction operation on the audio data, and a second-level reconstruction operation on the fusion vector and the dimensional vector, thereby finally completing the training of the audio reconstruction model. The finally trained audio reconstruction model is used to perform audio reconstruction operations on the reconstructed audio, thereby improving the accuracy and reliability of the obtained reconstructed audio. In addition, the reconstructed audio improves the richness of the audio material and improves the matching degree between the audio material and the adaptation content corresponding to the audio material, which is conducive to improving the audience's viewing experience of the reconstructed audio.

[0155] In an optional embodiment, if Figure 4 As shown, the second reconstruction module 303 may include a feature processing submodule 3031 and a reconstruction submodule 3032, wherein:

[0156] The feature processing submodule 3031 is used to perform a preset feature vector processing operation on the vector to be fused and the dimensional vector according to the preset vector dimension standard based on the audio reconstruction model, and obtain an audio processing vector corresponding to the vector to be fused, and the audio processing vector includes an audio identifier for representing the global information of the audio processing vector.

[0157] The reconstruction submodule 3032 is used to perform an audio reconstruction operation on the audio processing vector and the dimensional vector according to the audio reconstruction model and the audio identifier to obtain a secondary reconstruction result corresponding to the video to be processed.

[0158] In this optional embodiment, optionally, the feature processing submodule 3031 performs a preset feature vector processing operation on the to-be-fused vector and the dimension vector according to the preset vector dimension standard according to the audio reconstruction model, and obtains the audio processing vector corresponding to the to-be-fused vector in a manner specifically including:

[0159] According to the audio reconstruction model, vector splicing and dimension transformation operations are performed on the to-be-fused vector and the dimension vector at a preset dimension level to obtain a splicing transformation vector corresponding to the to-be-fused vector;

[0160] Input the splicing transformation vector into a preset fully connected layer to obtain an output vector corresponding to the splicing transformation vector;

[0161] The nonlinear transformation operation, dimensionality transformation operation, category labeling operation and mean conversion operation are sequentially performed on the output vector to obtain the mean conversion result corresponding to the output vector as the audio processing vector corresponding to the vector to be fused.

[0162] In this optional embodiment, optionally, the reconstruction submodule 3032 performs an audio reconstruction operation on the audio processing vector and the dimension vector according to the audio reconstruction model and the audio identifier, and obtains a secondary reconstruction result corresponding to the video to be processed. Specifically, the method includes:

[0163] According to the fully connected layer included in the audio reconstruction model, a feature fusion operation is performed on the audio processing vector to obtain a feature fusion result corresponding to the audio processing vector;

[0164] According to the audio identifier, a feature processing operation is performed on the feature fusion result to obtain a feature processing vector corresponding to the feature fusion result;

[0165] An audio loss parameter is calculated between the audio processing vector and the feature processing vector, and the audio loss parameter is used to determine the training progress of the audio reconstruction model.

[0166] Furthermore, the reconstruction submodule 3032 performs a feature processing operation on the feature fusion result according to the audio identifier, and obtains a feature processing vector corresponding to the feature fusion result in the following manner:

[0167] Obtaining non-fused vectors, where the non-fused vectors are other vectors in the first-level reconstruction result except the vector to be fused;

[0168] According to the audio identifier and the non-fused vector, a feature parsing operation is performed on the feature fusion result to obtain a parsing vector corresponding to the feature fusion result;

[0169] The audio identifier in the parsing vector is deleted, and the updated parsing vector is determined as the feature processing vector corresponding to the feature fusion result.

[0170] It can be seen that implementation Figure 4 The described audio reconstruction device for image-assisted audio completion can automatically fuse the vector to be processed and the dimensional vector according to the vector dimension standard based on the audio reconstruction model, thereby improving the accuracy of the reconstruction and fusion of audio data and image data; further, the audio processing vector obtained based on the pre-fusion operation is combined with the audio identifier and the dimensional vector, and input into the audio reconstruction model to perform reconstruction training of the audio reconstruction model, thereby improving the training efficiency of the audio reconstruction model and the accuracy and reliability of the obtained audio reconstruction model; in addition, the trained and accurate and reliable audio reconstruction model is used to reconstruct the audio to be reconstructed, thereby improving the accuracy of the obtained reconstructed audio, enriching the material content of the reconstructed audio, and also improving the matching degree between the reconstructed audio and the image content corresponding to the reconstructed audio, thereby improving the audience's viewing experience of the reconstructed audio (including images) to a certain extent.

[0171] Embodiment 4

[0172] See also Figure 5 , Figure 5FIG. 1 is a schematic diagram of the structure of another audio reconstruction device for image-assisted audio completion disclosed in an embodiment of the present invention. Figure 5 As shown, the audio reconstruction device of the image-assisted audio completion may include:

[0173] A memory 401 storing executable program codes;

[0174] a processor 402 coupled to the memory 401;

[0175] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the audio reconstruction method for image-assisted audio completion described in the first embodiment of the present invention or the second embodiment of the present invention.

[0176] Embodiment 5

[0177] An embodiment of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the steps in the audio reconstruction method of image-assisted audio completion described in Embodiment 1 or Embodiment 2 of the present invention.

[0178] Embodiment 6

[0179] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps in the audio reconstruction method of image-assisted audio completion described in Embodiment 1 or Embodiment 2.

[0180] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0181] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0182] Finally, it should be noted that the audio reconstruction method and device for image-assisted audio completion disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An audio reconstruction method for image-assisted audio completion, characterized in that: The method comprises: Performing a preprocessing operation on the acquired video to be processed according to a preset video preprocessing scheme to obtain separated data corresponding to the video to be processed, wherein the separated data includes audio data and image data, and the audio data is data obtained after at least one of a blank audio removal operation, a sampling rate conversion operation, and a feature extraction operation; Performing a primary reconstruction operation on the audio data according to a preset audio reconstruction model to obtain a primary reconstruction result corresponding to the audio data, wherein the primary reconstruction result includes a vector to be fused, and the vector to be fused is used to be fused with a dimensional vector corresponding to the image data to reconstruct the video to be processed, and the vector to be fused is consistent with the dimensional vector in feature dimension; According to the audio reconstruction model, a secondary reconstruction operation is performed on the vector to be fused and the dimensional vector to obtain a secondary reconstruction result corresponding to the video to be processed, wherein the secondary reconstruction result is used to determine an audio loss parameter of the audio reconstruction model, and the audio loss parameter is used to determine a training progress of the audio reconstruction model; When it is determined that the audio loss parameter indicates that the model convergence parameter of the audio reconstruction model is within a preset convergence threshold, it is determined that the training task for the audio reconstruction model is completed, and the trained audio reconstruction model is used to perform an audio reconstruction operation on the audio to be reconstructed; The preprocessing operation is performed on the acquired video to be processed according to the preset video preprocessing scheme to obtain the separated data corresponding to the video to be processed, including: According to a preset video preprocessing scheme, a preprocessing operation is performed on the acquired video to be processed to obtain initial audio data and image data corresponding to the video to be processed; Analyze the initial audio data to obtain audio parameters of the initial audio data, where the audio parameters include audio length and audio sampling rate; Determine whether the audio parameters of the initial audio data match the standard audio parameters, and when the determination result is yes, perform a convolution processing operation on the initial audio data according to a preset convolution layer to obtain first audio data corresponding to the initial audio data, and determine the first audio data and the image data as separated data corresponding to the video to be processed; When the judgment result is no, performing a parameter conversion operation on the initial audio data, and inputting the obtained parameter conversion result into a preset convolution layer to obtain second audio data corresponding to the initial audio data, and determining the second audio data and the image data as separated data corresponding to the video to be processed; And, performing convolution processing on the initial audio data or the parameter conversion result, comprising: Determine the target number of convolutional layers corresponding to the convolutional layer used to process the input data; According to the target number of convolution layers and a preset convolution processing scheme, a convolution operation is performed on the input data to obtain a convolution operation result corresponding to the input data, wherein the convolution operation result includes a convolution dimension value of the input data; Determine whether the convolution dimension value of the input data is greater than a preset dimension value, and when the judgment result is yes, perform a data cleaning operation on the convolution operation results that are greater than the preset dimension value to update the convolution operation results, and the convolution dimension value corresponding to the updated convolution operation result is equal to the preset dimension value; When it is determined that the convolution dimension value of the input data is less than the preset dimension value, vector completion is performed on the convolution operation result according to a preset dimension completion scheme so that the convolution dimension value in the convolution operation result is equal to the preset dimension value; After determining that the convolution dimension value corresponding to the convolution operation result is equal to the preset dimension value, the method further includes: Determining a target clustering value, where the target clustering value is used to perform a clustering operation on the convolution operation result; Performing a clustering operation on the convolution operation result to obtain a clustering result corresponding to the convolution operation result, wherein the clustering result includes a plurality of audio feature vectors, the number of all the audio feature vectors is the target clustering value, and each of the audio feature vectors corresponds to a plurality of sub-feature vectors; According to all the audio feature vectors, a target coding table is constructed, wherein the target coding table is used to convert a continuous audio vector into a discrete audio vector; For each audio vector in the convolution operation result, calculate the similarity parameter between the audio vector and each audio feature vector in the target coding table, and obtain a target parameter and a target audio vector corresponding to the target parameter, wherein the target parameter is a parameter with the smallest parameter value corresponding to the similarity parameter between the audio vector and each audio feature vector in the target coding table, and the target audio vector is a certain audio feature vector in the target coding table; For each audio vector in the convolution operation result, the audio vector is updated according to the target audio vector corresponding to the audio vector, so as to update the continuous audio vector in the convolution operation result to a discrete audio vector, and the updated convolution operation result is used to determine the loss function, and the loss function is used to determine the model convergence value of the subsequent audio reconstruction model.

2. The audio reconstruction method of image-assisted audio completion according to claim 1, characterized in that: The step of performing a secondary reconstruction operation on the vector to be fused and the dimensional vector according to the audio reconstruction model to obtain a secondary reconstruction result corresponding to the video to be processed includes: According to the audio reconstruction model and the preset vector dimension standard, a preset feature vector processing operation is performed on the vector to be fused and the dimension vector to obtain an audio processing vector corresponding to the vector to be fused, wherein the audio processing vector includes an audio identifier for representing global information of the audio processing vector; According to the audio reconstruction model and the audio identifier, an audio reconstruction operation is performed on the audio processing vector and the dimensional vector to obtain a secondary reconstruction result corresponding to the video to be processed.

3. The audio reconstruction method of image-assisted audio completion according to claim 2, characterized in that: The step of performing a preset feature vector processing operation on the vector to be fused and the dimensional vector according to the audio reconstruction model and a preset vector dimension standard to obtain an audio processing vector corresponding to the vector to be fused includes: According to the audio reconstruction model, performing vector splicing and dimensional transformation operations on the vector to be fused and the dimensional vector at a preset dimensional level to obtain a splicing transformation vector corresponding to the vector to be fused; Inputting the splicing transformation vector into a preset fully connected layer to obtain an output vector corresponding to the splicing transformation vector; The nonlinear transformation operation, the dimensional transformation operation, the category labeling operation and the mean conversion operation are sequentially performed on the output vector to obtain a mean conversion result corresponding to the output vector as the audio processing vector corresponding to the vector to be fused.

4. The audio reconstruction method of image-assisted audio completion according to claim 2, characterized in that: The step of performing an audio reconstruction operation on the audio processing vector and the dimensional vector according to the audio reconstruction model and the audio identifier to obtain a secondary reconstruction result corresponding to the video to be processed includes: According to the fully connected layer included in the audio reconstruction model, a feature fusion operation is performed on the audio processing vector to obtain a feature fusion result corresponding to the audio processing vector; According to the audio identifier, performing a feature processing operation on the feature fusion result to obtain a feature processing vector corresponding to the feature fusion result; An audio loss parameter between the audio processing vector and the feature processing vector is calculated, and the audio loss parameter is used to determine the training progress of the audio reconstruction model.

5. The audio reconstruction method of image-assisted audio completion according to claim 4, characterized in that: The performing a feature processing operation on the feature fusion result according to the audio identifier to obtain a feature processing vector corresponding to the feature fusion result includes: Acquire a non-fused vector, where the non-fused vector is other vectors in the first-level reconstruction result except the vector to be fused; According to the audio identifier and the non-fused vector, a feature parsing operation is performed on the feature fusion result to obtain a parsing vector corresponding to the feature fusion result; The audio identifier in the parsing vector is deleted, and the updated parsing vector is determined as the feature processing vector corresponding to the feature fusion result.

6. The audio reconstruction method of image-assisted audio completion according to any one of claims 1 to 5, characterized in that: The first-level reconstruction operation is specifically an audio mask operation, and performing the first-level reconstruction operation on the audio data according to a preset audio reconstruction model to obtain a first-level reconstruction result corresponding to the audio data includes: determining a target interval for performing the audio masking operation on the audio data; Setting a selection probability corresponding to each frame of data in the audio data; According to a preset audio reconstruction model, a selection probability corresponding to each frame of data, and the target interval, the audio mask operation is performed on the audio data to obtain a first-level reconstruction result corresponding to the audio data.

7. An audio reconstruction device for image-assisted audio completion, characterized in that: The device comprises: A preprocessing module, used to perform a preprocessing operation on the acquired video to be processed according to a preset video preprocessing scheme to obtain separated data corresponding to the video to be processed, wherein the separated data includes audio data and image data, and the audio data is data obtained after at least one of a blank audio removal operation, a sampling rate conversion operation, and a feature extraction operation; A first reconstruction module is used to perform a first-level reconstruction operation on the audio data according to a preset audio reconstruction model to obtain a first-level reconstruction result corresponding to the audio data, wherein the first-level reconstruction result includes a vector to be fused, and the vector to be fused is used to be fused with a dimensional vector corresponding to the image data to reconstruct the video to be processed, and the vector to be fused is consistent with the dimensional vector in feature dimension; A second reconstruction module is used to perform a secondary reconstruction operation on the vector to be fused and the dimensional vector according to the audio reconstruction model to obtain a secondary reconstruction result corresponding to the video to be processed, wherein the secondary reconstruction result is used to determine an audio loss parameter of the audio reconstruction model, and the audio loss parameter is used to determine a training progress of the audio reconstruction model; A determination module, configured to determine that the training task for the audio reconstruction model is completed when it is determined that the audio loss parameter indicates that the model convergence parameter of the audio reconstruction model is within a preset convergence threshold, and the trained audio reconstruction model is used to perform an audio reconstruction operation on the audio to be reconstructed; The preprocessing module performs a preprocessing operation on the video to be processed according to a preset video preprocessing scheme, and obtains the separated data corresponding to the video to be processed in a manner specifically including: According to a preset video preprocessing scheme, a preprocessing operation is performed on the video to be processed to obtain initial audio data and image data corresponding to the video to be processed; Analyze the initial audio data to obtain audio parameters of the initial audio data, where the audio parameters include audio length and audio sampling rate; Determine whether the audio parameters of the initial audio data match the standard audio parameters, and when the determination result is yes, perform a convolution processing operation on the initial audio data according to a preset convolution layer to obtain first audio data corresponding to the initial audio data, and determine the first audio data and the image data as separated data corresponding to the video to be processed; When the judgment result is no, performing a parameter conversion operation on the initial audio data, and inputting the obtained parameter conversion result into a preset convolution layer to obtain second audio data corresponding to the initial audio data, and determining the second audio data and the image data as separated data corresponding to the video to be processed; The device is also used for: The manner of performing convolution processing on the initial audio data or the parameter conversion result specifically includes: Determine the target number of convolutional layers corresponding to the convolutional layer used to process the input data; According to the target number of convolution layers and a preset convolution processing scheme, a convolution operation is performed on the input data to obtain a convolution operation result corresponding to the input data, wherein the convolution operation result includes a convolution dimension value of the input data; Determine whether the convolution dimension value of the input data is greater than a preset dimension value, and when the judgment result is yes, perform a data cleaning operation on the convolution operation results that are greater than the preset dimension value to update the convolution operation results, and the convolution dimension value corresponding to the updated convolution operation result is equal to the preset dimension value; When it is determined that the convolution dimension value of the input data is less than the preset dimension value, vector completion is performed on the convolution operation result according to a preset dimension completion scheme so that the convolution dimension value in the convolution operation result is equal to the preset dimension value; The device is also used for: After determining that the convolution dimension value corresponding to the convolution operation result is equal to the preset dimension value, determining a target clustering value, where the target clustering value is used to perform a clustering operation on the convolution operation result; Performing a clustering operation on the convolution operation result to obtain a clustering result corresponding to the convolution operation result, wherein the clustering result includes a plurality of audio feature vectors, the number of all the audio feature vectors is the target clustering value, and each of the audio feature vectors corresponds to a plurality of sub-feature vectors; According to all the audio feature vectors, a target coding table is constructed, wherein the target coding table is used to convert a continuous audio vector into a discrete audio vector; For each audio vector in the convolution operation result, calculate the similarity parameter between the audio vector and each audio feature vector in the target coding table, and obtain a target parameter and a target audio vector corresponding to the target parameter, wherein the target parameter is a parameter with the smallest parameter value corresponding to the similarity parameter between the audio vector and each audio feature vector in the target coding table, and the target audio vector is a certain audio feature vector in the target coding table; For each audio vector in the convolution operation result, the audio vector is updated according to the target audio vector corresponding to the audio vector, so as to update the continuous audio vector in the convolution operation result to a discrete audio vector, and the updated convolution operation result is used to determine the loss function, and the loss function is used to determine the model convergence value of the subsequent audio reconstruction model.

8. An audio reconstruction device for image-assisted audio completion, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the audio reconstruction method for image-assisted audio completion as described in any one of claims 1-6.

9. A computer storage medium, characterized in that The computer storage medium stores computer instructions, which, when called, are used to execute the audio reconstruction method for image-assisted audio completion as described in any one of claims 1-6.

Images