Audio information generation method, device, equipment and medium based on image analysis

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for generating audio information based on image analysis. The method extracts N clipped images from an image to be converted by sliding a window, inputs the N clipped images into a temporal convolutional network model, outputs pseudo-temporal features, inputs the pseudo-temporal features and image features into an encoder to obtain compressed features, reconvolves the compressed features to obtain reconvolved features, searches for audio features most similar to the reconvolved features in a preset audio feature set to obtain target audio features, generates reference audio features based on the target audio features, reconstructs the reference audio features to obtain generated audio information, clips the image to be converted by sliding, so that the clipped images contain richer image information, improves feature representation capabilities, maps the reconvolved features into target audio features that conform to audio conventions, and improves the correlation between the image and the generated audio.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for generating audio information based on image analysis. Background Art

[0002] At present, in many application fields such as game voice-overs, short video soundtracks, and metaverse background sounds, image signals and audio signals have a strong correlation and can be combined to provide users with a better audio-visual experience. The existing image-based music matching method still mainly uses manual music matching, that is, providing image music through human composition.

[0003] However, manual music creation requires significant creative effort. Deep learning approaches for audio generation typically involve converting text to audio, such as Google's Noise2music model. This approach can generate audio that matches a pre-defined theme. However, when applied to image-to-audio conversion, images contain far more information than text, making it difficult to describe the image's information in detail based solely on the theme. This results in a low correlation between the image and the generated audio. Therefore, improving the correlation between images and generated audio has become a pressing issue. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device, and medium for generating audio information based on image analysis to solve the problem of low correlation between images and generated audio.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating audio information based on image analysis, the method comprising:

[0006] Obtain an image to be converted and its image features, and perform sliding extraction in the image to be converted using a preset sliding window according to a set step size to obtain N intercepted images, where N is an integer greater than zero;

[0007] Input the N intercepted images into the trained temporal convolutional network model to extract temporal features, and output pseudo temporal features corresponding to the image to be converted;

[0008] Inputting the pseudo time series features and the image features into a trained encoder for feature compression to obtain compressed features, and reconvolving the compressed features using a convolutional layer to obtain reconvolution features corresponding to at least one convolution kernel in the convolutional layer;

[0009] For any reconvolution feature, searching for an audio feature that is most similar to the reconvolution feature in a preset audio feature set to obtain a target audio feature corresponding to the reconvolution feature;

[0010] Based on all target audio features, reference audio features are generated, feature reconstruction is performed on the reference audio features, and the feature reconstruction results are determined as generated audio information corresponding to the image to be converted.

[0011] In a second aspect, an embodiment of the present invention provides an audio information generation device based on image analysis, the audio information generation device comprising:

[0012] An image capture module is used to obtain an image to be converted and its image features, and to perform sliding extraction in the image to be converted using a preset sliding window according to a set step size to obtain N captured images, where N is an integer greater than zero;

[0013] A first feature extraction module is used to input the N intercepted images into a trained temporal convolutional network model to extract temporal features and output pseudo temporal features corresponding to the image to be converted;

[0014] A second feature extraction module is configured to input the pseudo time series features and the image features into a trained encoder for feature compression to obtain compressed features, and reconvolute the compressed features using a convolutional layer to obtain reconvolution features corresponding to at least one convolution kernel in the convolutional layer;

[0015] A feature search module is configured to search, for any reconvolution feature, a preset audio feature set for an audio feature that is most similar to the reconvolution feature, and obtain a target audio feature corresponding to the reconvolution feature;

[0016] The audio generation module is used to generate reference audio features based on all target audio features, perform feature reconstruction on the reference audio features, and determine the feature reconstruction results as generated audio information corresponding to the image to be converted.

[0017] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the audio information generation method as described in the first aspect when executing the computer program.

[0018] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the audio information generation method as described in the first aspect is implemented.

[0019] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0020] Obtain the image to be converted and its image features, perform sliding extraction in the image to be converted according to the set step size through a preset sliding window to obtain N cropped images, input the N cropped images into the trained temporal convolutional network model for temporal feature extraction, output the pseudo temporal features corresponding to the image to be converted, input the pseudo temporal features and image features into the trained encoder for feature compression to obtain compressed features, use the convolution layer to reconvolve the compressed features, obtain the reconvolution features corresponding to at least one convolution kernel in the convolution layer, and for any reconvolution feature, search the preset audio feature set for the audio feature that is most similar to the reconvolution feature. Frequency features are obtained to obtain target audio features corresponding to the reconvolution features. Reference audio features are generated based on all target audio features. Feature reconstruction is performed on the reference audio features to determine the feature reconstruction result as the generated audio information corresponding to the image to be converted. The image to be converted is intercepted in a sliding manner so that the intercepted image contains different image information, thereby extracting pseudo-time series features. Through audio feature search, the reconvolution features can be mapped to target audio features that conform to audio habits, thereby generating audio, improving the coupling of image features and audio features, and thereby improving the correlation between the image and the generated audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 This is a schematic diagram of an application environment of an audio information generation method based on image analysis provided in the first embodiment of the present invention;

[0023] Figure 2 This is a flow chart of a method for generating audio information based on image analysis provided in the first embodiment of the present invention;

[0024] Figure 3 1 is a schematic structural diagram of an audio information generation device based on image analysis provided in a second embodiment of the present invention;

[0025] Figure 4 This is a structural diagram of a computer device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0026] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0028] It will also be understood that the term "and / or" used in the present description and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0029] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0030] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" in the present specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present invention. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] Embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0033] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] It should be understood that the order of execution of the steps in the following embodiments does not necessarily mean the order in which they are executed. The order in which each process is executed should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0035] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0036] The first embodiment of the present invention provides an audio information generation method based on image analysis, which can be applied in the following situations: Figure 1 In an application environment, clients communicate with servers. Clients include, but are not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud-based terminal devices, personal digital assistants (PDAs), and other computer devices. Servers can be standalone servers or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0037] See also Figure 2 , is a flow chart of a method for generating audio information based on image analysis provided by the first embodiment of the present invention. The above-mentioned method for generating audio information based on image analysis can be applied to Figure 1The server in the server, the computer device corresponding to the server communicates with the client to obtain the image to be converted and its image features provided by the client. The image to be converted can refer to the image that needs to generate audio information. The image features can be used to characterize the spatial information of the image to be converted. The computer device corresponding to the server is deployed with a trained temporal convolutional network model, a trained encoder and a convolutional layer. The trained temporal convolutional network model can be used to extract temporal features, the trained encoder can be used for feature compression, and the convolutional layer can be used for feature reconvolution processing. Figure 2 As shown, the audio information generation method based on image analysis may include the following steps:

[0038] Step S201 : obtaining an image to be converted and its image features, performing sliding extraction in the image to be converted using a preset sliding window according to a set step size, and obtaining N captured images.

[0039] Among them, the image to be converted can refer to an image that needs to generate audio information, the image to be converted can be input into the client by the user, the image feature can represent the spatial feature information of the image to be converted, the preset sliding window can refer to a sliding window, the preset sliding window can be used to capture an image from the image to be converted, the set step size can refer to the sliding step size of the preset sliding window, the captured image can refer to the image captured when the preset sliding window slides on the image to be converted, and N is an integer greater than zero.

[0040] Specifically, the image to be converted needs to meet the preset size, and the preset size is set to W*H, where W is the width of the image to be converted and H is the height of the image to be converted. For example, the preset size can be set to 1024*648, and the size of the preset sliding window also meets the preset size, which is also W*H.

[0041] Optionally, performing sliding extraction in the image to be converted according to a set step size through a preset sliding window to obtain N captured images includes:

[0042] Set the sliding direction, slide the preset sliding window according to the sliding direction and the set step size, retain all pixels in the intersection of the preset sliding window and the image to be converted, and set other pixels to zero to obtain the intercepted image;

[0043] Return to the step of sliding the preset sliding window according to the sliding direction and the set step size, retaining all pixels in the intersection of the sliding preset sliding window and the image to be converted, and setting other pixels to zero to obtain the intercepted image. The iteration stops when the intersection of the sliding preset sliding window and the image to be converted is an empty set, and N intercepted images are obtained.

[0044] The sliding direction may refer to a preset sliding direction of the sliding window.

[0045] Specifically, the initial position of the preset sliding window can be located on the left side of the image to be converted, so that the rightmost edge of the preset sliding window coincides with the leftmost edge of the image to be converted. At this time, the sliding direction is to the right, and the preset sliding window is slid according to the preset step size to extract N captured images.

[0046] The value of the set step size can be an integer greater than or equal to 1 and less than W. In this embodiment, the value of the set step size is set to 16. For an image to be converted with a width W of 1024, the preset sliding window can be completely overlapped with the image to be converted by sliding it to the right 64 times. The leftmost edge of the preset sliding window can be overlapped with the rightmost edge of the image to be converted by sliding it to the right 128 times. In this way, a total of 127 captured images can be obtained, that is, N is 127.

[0047] The element values ​​of all elements in the preset sliding window can be set to 1. When the preset sliding window slides at any time, each element in the preset sliding window is multiplied by the element at the corresponding position in the image to be converted. If any element in the preset sliding window does not have an element at a corresponding position in the image to be converted, the element in the preset sliding window is multiplied by zero, and the preset sliding window after the element value multiplication is determined to be the intercepted image and a copy of the intercepted image is generated. The preset sliding window is initialized, that is, the element values ​​of all elements in the preset sliding window are set to 1, and the next sliding is performed until the intersection of the preset sliding window and the image to be converted after sliding is an empty set. At this time, each element in the preset sliding window does not have an element at a corresponding position in the image to be converted.

[0048] In one embodiment, the initial position of the preset sliding window can also be located in various directions such as the right side, top side, bottom side, and upper left side of the image to be converted. It is only necessary to set the sliding direction so that the preset sliding window can intersect with the image to be converted during the sliding process. For example, when the initial position of the preset sliding window is located on the right side of the image to be converted, the leftmost edge of the preset sliding window is made to coincide with the rightmost edge of the image to be converted. In this case, the sliding direction can be to the left. The preset sliding window is slid according to a preset step size to extract N captured images.

[0049] In this embodiment, the cropped images are extracted by sliding a preset sliding window, so that the image information contained in the cropped images is different, and thus when feature extraction is performed subsequently, the model can extract different features according to different cropped images. For example, for a landscape painting image, there is water at the bottom of the image and mountains at the top. When the preset sliding window slides from bottom to top, the first cropped image extracted only includes water information, and then a cropped image containing both mountain and water information is extracted, and finally only a cropped image containing mountain information is extracted. Each cropped image contains different information, and the model's perception focus is also different when extracting features, thereby effectively enriching the data input of the model.

[0050] The above-mentioned steps of obtaining the image to be converted and its image features, performing sliding extraction in the image to be converted according to a set step size through a preset sliding window, and obtaining N captured images can make the subsequently extracted image features richer and more complete, thereby improving the accuracy of generating audio information based on the image, that is, improving the correlation between the image and the generated audio.

[0051] In step S202 , N cropped images are input into a trained temporal convolutional network model to extract temporal features, and pseudo temporal features corresponding to the image to be converted are output.

[0052] Among them, the input of the trained temporal convolutional network model is data that conforms to the time sequence, and the output of the trained temporal convolutional network model is the temporal feature. The pseudo-temporal feature can be used to characterize the temporal features of N captured images.

[0053] Specifically, since images do not contain information in the time dimension, if it is desired to generate audio information containing timing information through images, the image information needs to be forged into a set of images with timing information. The image set is N captured images, which are extracted by sliding a preset window. They can approximately represent the process of gradually moving the field of view from one side of the image to the other side, and this process can be considered to be in line with the time sequence, that is, the N captured images can be approximately considered to be in line with the time sequence.

[0054] In one embodiment, the implementer may directly use images of all frames in a continuous video segment as the N captured images. Under the constraint of the continuous video segment, the N captured images must be in a time sequence.

[0055] In one embodiment, the implementer may first convert the image to be converted into data with temporal information, such as a text description of the image, and then directly extract temporal features from the text description through a trained temporal convolutional network model, thereby also obtaining pseudo-temporal features corresponding to the image to be converted.

[0056] The trained temporal convolutional network model can include multiple hidden layers. The temporal convolutional network model uses a slider to determine a single input, and regards a single captured image as an input node. The size of the slider can include multiple input nodes. For example, the slider can include three input nodes. The movement of the slider also needs to set the corresponding step size, and the movement direction can be from left to right. In this embodiment, the step size can be set to 2, then the first input is [1, 2, 3], and the numbers can be used to identify the input node, that is, the position of the captured image in the sequence of N captured images. Since the step size is set to 2, the second input is [3, 4, 5], and so on.

[0057] The first hidden layer extracts the temporal features of multiple cropped images input each time to obtain the first sub-feature. When processing in the second hidden layer, the first sub-feature is used as the input node and slides with the same slider size, moving direction and step size. The second hidden layer extracts the temporal features of multiple first sub-features input each time to obtain the second sub-feature.

[0058] As the number of hidden layers increases, each hidden layer uses the output of the previous hidden layer as an input node, and a slider is used to determine a single input until the number of hidden layers reaches the preset layer. At this time, only one input can be determined by the slider, and the time series features of the input are extracted to obtain the final integrated features, which are also the pseudo time series features of the N captured images.

[0059] It should be noted that in this embodiment, since the final output is a pseudo-time series feature, only the encoder part of the trained time convolutional network model is needed. In order to ensure the effectiveness of the encoder part in extracting time series features, during the training process of the time convolutional network model, a classifier can be connected behind the encoder part, and the training sample is input into the encoder part for feature extraction to obtain predicted time series features. The predicted time series features are then input into the classifier, and the predicted classification results are output. The loss function is calculated based on the predicted classification results and the labels of the corresponding training samples to obtain the training loss. The parameters of the encoder part and the classifier are adjusted according to the training loss until the training loss converges to obtain the trained encoder part. The loss function can use a cross-entropy loss function. The label can refer to an emotional label. The emotional label can represent the image of the image, and can also represent the emotional information of the audio information, which is suitable as a label in this embodiment. The emotional label can include a happy label, a sad label, a fear label, etc.

[0060] Optionally, N captured images are input into a trained temporal convolutional network model to extract temporal features, and the output pseudo-temporal features corresponding to the image to be converted include:

[0061] According to the extraction order, an image sequence is formed by N intercepted images;

[0062] The image sequence is input into the trained temporal convolutional network model to extract temporal features and output pseudo temporal features.

[0063] The extraction order may refer to the order in which the captured images are captured, and the image sequence may include N captured images that conform to the extraction order.

[0064] Specifically, the extraction order can be approximately regarded as the time order, so that the N cropped images contained in the image sequence approximately conform to the time order, so that the time series features can be extracted through the trained time series convolutional network model. At this time, the extracted features are pseudo-time features.

[0065] The above-mentioned step of inputting N cropped images into the trained temporal convolutional network model for temporal feature extraction and outputting pseudo temporal features corresponding to the image to be converted can effectively extract the temporal features of the cropped images, thereby facilitating the provision of temporal information for audio information generation and increasing the degree of correlation between the image and the generated audio information.

[0066] In step S203, the pseudo time series features and the image features are input into the trained encoder for feature compression to obtain compressed features, and the compressed features are reconvolved using a convolutional layer to obtain reconvolution features corresponding to at least one convolution kernel in the convolutional layer.

[0067] Among them, the trained encoder can be used to further compress the pseudo-time series features and image features. The compressed features can represent the spatiotemporal features of the image to be converted. The convolution layer can be used to further extract the compressed features, that is, re-convolution. The convolution layer can include at least one convolution kernel, and one convolution kernel corresponds to a convolution result, which is also the re-convolution feature.

[0068] Specifically, the trained encoder can include two encoding branches, the input of one encoding branch is pseudo-time series features, and the input of the other encoding branch is image features. The purpose of setting two encoding branches is to solve the problem of inconsistent sizes of pseudo-time series features and image features, constrain the output sizes of the two encoding branches to be consistent, and add the encoding features obtained by the two encoding branches point by point to determine the point-by-point addition result as the compressed feature.

[0069] Since the compressed features are used to characterize the spatiotemporal features of the image to be converted, the spatiotemporal features should be correlated with the audio information. Therefore, during the training process of the encoder, an additional encoding branch of the audio information can be added, and the encoding branch of the audio information can be trained. Specifically, the sample acquisition in the training process can be extracted from scenes such as game scenes and short video scenes with soundtracks. For a training image sample, there is an associated audio information sample. The training image sample is input into the first encoding branch and the second encoding branch to obtain image sample features and pseudo-time sample features. The audio information sample is input into the third encoding branch to obtain audio sample features. The image sample features and pseudo-time sample features are weighted added for the first time to obtain compressed sample features. The weights of the first weighted addition can be evenly distributed. It is set to 1, and the implementer can adjust it according to the actual situation, and then perform a second weighted addition on the compressed sample features and the audio sample features. In this embodiment, the weights of the second weighted addition can all be set to 0.5, and the implementer can adjust it according to the actual situation. The second weighted addition result is input into the temporary classifier, and the reference classification result is output. According to the labels corresponding to the training image samples and their associated audio information samples and the reference classification results, the first encoding branch and the second encoding branch are trained to obtain a trained encoder. The label can still be an emotional label. In the training process of the encoder, the association between audio information, image features and pseudo-time series features is directly modeled. Compared with the training of the above-mentioned time convolutional network model, the correlation between image information and audio information is further enhanced.

[0070] Optionally, the compressed features are reconvolved using a convolution layer, and the reconvolution features corresponding to at least one convolution kernel in the convolution layer include:

[0071] For any convolution kernel, the compressed feature and the convolution kernel are convolved to obtain the reconvolution feature corresponding to the convolution kernel;

[0072] Traverse all convolution kernels and obtain the reconvolution features corresponding to each convolution kernel in the convolution layer.

[0073] Among them, the convolution kernel can be used for convolution operation. In this embodiment, the compressed feature can represent the spatiotemporal information. The number of convolution kernels is set to be consistent with the time dimension of the compressed feature. At this time, the reconvolution feature corresponding to each convolution kernel can represent the deep features of each moment.

[0074] Specifically, the training of the convolutional layer can be supervised by real audio features, and the training samples can be compressed sample features. The audio samples associated with the images corresponding to the compressed sample features are obtained. According to the number of convolution kernels, the audio samples are framed and processed to obtain the same number of frame results as the number of convolution kernels. The frame features corresponding to each frame result are extracted, and the frame features can be used as real audio features. The similarity between the reconvolution feature and the frame feature is calculated as the loss function for convolution layer training. The convolution layer is trained to obtain a trained convolution layer.

[0075] Step S204: for any reconvolution feature, search for an audio feature that is most similar to the reconvolution feature in a preset audio feature set to obtain a target audio feature corresponding to the reconvolution feature.

[0076] Among them, the preset audio feature set may include several audio features. The audio feature may refer to the feature corresponding to the audio information that conforms to the audio habits. The audio information that conforms to the audio habits may be chords, etc. The target audio feature may refer to the audio feature that is closest to the reconvolution feature.

[0077] Specifically, since this embodiment needs to convert the image into audio information, the conventional method of extracting image features for reconstruction will inevitably result in a large amount of noise, resulting in poor audio information generation. Therefore, this embodiment presets an audio feature set. Before feature reconstruction, the reconvolution feature needs to be searched in the audio feature set to obtain the audio feature that is most similar to the reconvolution feature, that is, the target audio feature. The target audio feature can not only ensure the retention of image features, but also ensure that the audio features used for reconstruction conform to audio information habits.

[0078] The similarity between the audio feature and the reconvolution feature can be measured by cosine similarity, Euclidean distance, etc. In this embodiment, cosine similarity is used to calculate the similarity, and the value range of the calculation result is [0, 1]. The closer the calculation result is to 1, the greater the similarity between the audio feature and the reconvolution feature. The closer the calculation result is to 0, the smaller the similarity between the audio feature and the reconvolution feature. For any reconvolution feature, the similarity between the reconvolution feature and each audio feature in the audio feature set is calculated, and the audio feature corresponding to the maximum similarity is determined as the target audio feature of the corresponding reconvolution feature.

[0079] The above-mentioned step of searching for the audio feature most similar to the reconvolution feature in the preset audio feature set for any reconvolution feature to obtain the target audio feature corresponding to the reconvolution feature, and obtaining the target audio feature corresponding to the reconvolution feature by searching, can not only ensure the retention of image features, but also ensure that the audio features used for reconstruction conform to the audio information habits, thereby improving the accuracy of subsequent audio information generation.

[0080] Step S205 : generating reference audio features based on all target audio features, performing feature reconstruction on the reference audio features, and determining the feature reconstruction results as generated audio information corresponding to the image to be converted.

[0081] The reference audio features may include all target audio features, the reference audio features may be used to reconstruct the audio information, and the generated audio information may refer to the generated result of the audio information.

[0082] Specifically, feature reconstruction can be achieved using a trained decoder, and the training process of the decoder can adopt a pre-training method, that is, audio feature samples are used to form a training data set, and the audio information corresponding to the audio feature samples is used as a label. The decoder is trained based on the generated sample audio information and labels obtained by inputting the audio feature samples into the decoder to obtain a trained decoder.

[0083] Optionally, generating reference audio features according to all target audio features includes:

[0084] According to the arrangement order of all convolution kernels, all target audio features searched based on the reconvolution features of the corresponding convolution kernels are spliced ​​to obtain a splicing result;

[0085] The splicing result is determined as the reference audio feature.

[0086] Among them, the arrangement order of all convolution kernels can refer to the arrangement order of the corresponding time dimension, the splicing can be performed in a concatenated manner, and the splicing result can refer to the splicing result of all reconvolution features. Since the generation of audio information usually requires the generation of multiple frames of audio rather than a single frame of audio, it is necessary to determine the corresponding target audio features for each frame of audio.

[0087] Optionally, after generating reference audio features based on all target audio features, the method further includes:

[0088] Input the reference audio features into the trained discriminator and output the identification results;

[0089] If the identification result is passed, the step of reconstructing the reference audio features and determining the feature reconstruction result as generated audio information corresponding to the image to be converted is executed.

[0090] Among them, the trained discriminator can be used to identify whether the reference audio feature is a noise feature. The identification result can include pass and fail. Pass can indicate that the reference audio feature is not a noise feature, and fail can indicate that the reference audio feature is a noise feature.

[0091] Specifically, the training of the discriminator can use noisy audio features and non-noise audio features as training data, and the training data can be labeled, and the labeling categories are also passed and failed. It should be noted that in the labeling process, non-noise audio features do not mean that there is no noise at all. As long as the audio information corresponding to the training data can conform to normal audio habits, it is considered to be a non-noise audio feature. For example, the audio information corresponding to the training data is a melody, but contains slight white noise. At this time, the training data is still considered to be labeled as a non-noise audio feature.

[0092] If the identification result is passed, it means that the reference audio feature contains valid audio information at this time, and the step of reconstructing the reference audio feature and determining that the feature reconstruction result is the generated audio information corresponding to the image to be converted is executed.

[0093] Optionally, performing feature reconstruction on the reference audio feature and determining the feature reconstruction result as generated audio information corresponding to the image to be converted includes:

[0094] Input the reference audio features into the trained melody decoder to reconstruct the melody and output the first sub-audio information;

[0095] Input the reference audio features into the trained denoising decoder to reconstruct the denoised audio and output the second sub-audio information;

[0096] The average of the first sub audio information and the second sub audio information is used as the generated audio information.

[0097] Among them, the trained melody decoder is used for melody reconstruction, that is, it pays more attention to the melody and time correlation of the generated audio information, but ignores the quality of the generated audio information, that is, the audio information generated by the trained melody decoder may contain certain noise. The first sub-audio information can refer to the result of melody reconstruction in the trained melody decoder input with reference audio features.

[0098] The trained denoising decoder is used for denoised frequency reconstruction, that is, it pays more attention to the quality of the generated audio information, but ignores the melody of the generated audio information, that is, the audio information generated by the trained denoising decoder may not have a melody. The first sub-audio information can refer to the result of denoised frequency reconstruction in the trained denoising decoder by inputting the reference audio features.

[0099] Specifically, mean calculation is essentially a special case of weighted calculation. At this time, the weights of the weighted calculation of the first sub-audio information and the second sub-audio information are both 0.5. The implementer can adjust the weighting according to the actual needs of generating audio information. For example, if more attention is paid to the melody part, the weight of the first sub-audio information can be increased, and the weight of the second sub-audio information can be reduced. They can be set to 0.6 and 0.4 respectively.

[0100] The above steps of generating reference audio features based on all target audio features, reconstructing the reference audio features, and determining the feature reconstruction results as the generated audio information corresponding to the image to be converted, because the target audio features are used to form the reference audio features, the generated audio information obtained by reconstructing the reference audio features is more in line with audio habits, thereby improving the audio quality and the degree of correlation between the generated audio and the image.

[0101] In this embodiment, through audio feature search, the reconvolution features can be mapped to target audio features that conform to audio habits, thereby generating audio, improving the coupling between image features and audio features, and further improving the correlation between the image and the generated audio.

[0102] Corresponding to the audio information generation method based on image analysis of the above embodiment, Figure 3 The structural block diagram of the audio information generation device based on image analysis provided in the second embodiment of the present invention is shown. The above-mentioned audio information generation device based on image analysis is applied to the server side. The computer device corresponding to the server side communicates with the client side to obtain the image to be converted and its image features provided by the client side. The image to be converted may refer to the image for which audio information generation is required. The image features can be used to characterize the spatial domain information of the image to be converted. The computer device corresponding to the server side is deployed with a trained temporal convolutional network model, a trained encoder and a convolutional layer. The trained temporal convolutional network model can be used to extract temporal features, the trained encoder can be used for feature compression, and the convolutional layer can be used for feature reconvolution processing. For ease of explanation, only the parts related to the embodiment of the present invention are shown.

[0103] See also Figure 3 , the audio information generation device based on image analysis includes:

[0104] An image capture module 31 is configured to obtain an image to be converted and its image features, and to perform sliding extraction in the image to be converted using a preset sliding window according to a set step size to obtain N captured images, where N is an integer greater than zero;

[0105] The first feature extraction module 32 is used to input N intercepted images into the trained temporal convolutional network model to extract temporal features and output pseudo temporal features corresponding to the image to be converted;

[0106] A second feature extraction module 33 is configured to input the pseudo time series features and the image features into a trained encoder for feature compression to obtain compressed features, and to reconvolute the compressed features using a convolutional layer to obtain reconvolution features corresponding to at least one convolution kernel in the convolutional layer;

[0107] A feature search module 34 is configured to search, for any reconvolution feature, a preset audio feature set for an audio feature that is most similar to the reconvolution feature, and obtain a target audio feature corresponding to the reconvolution feature;

[0108] The audio generation module 35 is configured to generate reference audio features based on all target audio features, perform feature reconstruction on the reference audio features, and determine the feature reconstruction results as generated audio information corresponding to the image to be converted.

[0109] Optionally, the image capture module 31 includes:

[0110] A sliding window unit is used to set a sliding direction, slide a preset sliding window according to the sliding direction and the set step size, retain all pixels in the intersection of the preset sliding window and the image to be converted after sliding, and set other pixels to zero to obtain a cut-off image;

[0111] The iterative sliding unit is used to return to the step of sliding the preset sliding window according to the sliding direction and the set step size, retaining all pixels in the intersection of the sliding preset sliding window and the image to be converted, and setting other pixels to zero to obtain a cut-off image, until the intersection of the sliding preset sliding window and the image to be converted is an empty set, stopping the iteration, and obtaining N cut-off images.

[0112] Optionally, the first feature extraction module 32 includes:

[0113] a sequence forming unit, configured to form an image sequence from the N captured images in an extraction order;

[0114] The temporal feature extraction unit is used to input the image sequence into the trained temporal convolutional network model to extract temporal features and output pseudo temporal features.

[0115] Optionally, the second feature extraction module 33 includes:

[0116] The convolution calculation unit is used to perform convolution calculation on the compressed features and the convolution kernel for any convolution kernel to obtain the reconvolution features corresponding to the convolution kernel;

[0117] The convolution kernel traversal unit is used to traverse all convolution kernels and obtain the reconvolution features corresponding to each convolution kernel in the convolution layer.

[0118] Optionally, the audio generation module 35 includes:

[0119] A feature splicing unit is used to splice all target audio features searched based on the reconvolution features of the corresponding convolution kernels according to the arrangement order of all convolution kernels to obtain a splicing result;

[0120] The reference feature determination unit is used to determine the splicing result as a reference audio feature.

[0121] Optionally, the audio information generating device further includes:

[0122] The feature identification module is used to input the reference audio features into the trained discriminator and output the identification results;

[0123] The step execution module is used to execute the step of reconstructing the reference audio features and determining the feature reconstruction result as the generated audio information corresponding to the image to be converted if the identification result is passed.

[0124] Optionally, the audio generation module 35 includes:

[0125] A first reconstruction unit is configured to input the reference audio feature into a trained melody decoder to reconstruct the melody and output first sub-audio information;

[0126] The second reconstruction unit is used to input the reference audio feature into the trained denoising decoder to perform denoised audio reconstruction and output the second sub-audio information;

[0127] The mean value calculation unit is used to generate audio information using the mean value of the first sub-audio information and the second sub-audio information.

[0128] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules and units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0129] Figure 4 This is a schematic diagram of the structure of a computer device provided in the third embodiment of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned embodiments of the method for generating audio information based on image analysis are implemented.

[0130] The computer device may include, but is not limited to, a processor and a memory. It will be understood by those skilled in the art that Figure 4 The above is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0131] The processor may be a CPU, other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0132] Memory includes readable storage media, internal memory, and the like. Internal memory can be the internal memory of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage medium. The readable storage medium can be the computer device's hard drive. In other embodiments, it can also be an external storage device, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, or a flash memory card. Furthermore, memory can include both the computer device's internal storage unit and external storage devices. Memory is used to store the operating system, application programs, boot loaders, data, and other programs, such as the program code of computer programs. Memory can also be used to temporarily store data that has been output or is about to be output.

[0133] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. Computer-readable media can include at least: any entity or device capable of carrying computer program code, recording media, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunications signals, and software distribution media. Examples include USB flash drives, removable hard drives, magnetic disks, or optical disks. In some jurisdictions, based on legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunications signals.

[0134] The present invention may implement all or part of the processes in the above-mentioned method embodiments, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0135] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0136] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0137] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0138] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0139] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A method for generating audio information based on image analysis, characterized in that: The audio information generation method comprises: Obtain an image to be converted and its image features, and perform sliding extraction in the image to be converted using a preset sliding window according to a set step size to obtain N intercepted images, where N is an integer greater than zero; Input the N intercepted images into the trained temporal convolutional network model to extract temporal features, and output pseudo temporal features corresponding to the image to be converted; Inputting the pseudo time series features and the image features into a trained encoder for feature compression to obtain compressed features, and reconvolving the compressed features using a convolutional layer to obtain reconvolution features corresponding to at least one convolution kernel in the convolutional layer; For any reconvolution feature, searching for an audio feature that is most similar to the reconvolution feature in a preset audio feature set to obtain a target audio feature corresponding to the reconvolution feature; Based on all target audio features, reference audio features are generated, feature reconstruction is performed on the reference audio features, and the feature reconstruction results are determined as generated audio information corresponding to the image to be converted.

2. The audio information generating method according to claim 1, wherein: The step of performing sliding extraction on the image to be converted by using a preset sliding window according to a set step size to obtain N intercepted images includes: Setting a sliding direction, sliding the preset sliding window according to the sliding direction and the set step size, retaining all pixels in the intersection of the preset sliding window and the image to be converted within the preset sliding window, and setting other pixels to zero, to obtain the captured image; Returning to the step of sliding the preset sliding window according to the sliding direction and the set step size, retaining all pixels in the intersection of the sliding preset sliding window and the image to be converted, setting other pixels to zero, and obtaining the captured image, until the intersection of the sliding preset sliding window and the image to be converted is an empty set, stopping iteration, and obtaining the N captured images.

3. The audio information generating method according to claim 1, wherein: Inputting the N intercepted images into the trained temporal convolutional network model to extract temporal features, and outputting pseudo temporal features corresponding to the image to be converted includes: forming an image sequence from the N captured images in an extraction order; The image sequence is input into the trained temporal convolutional network model to extract temporal features, and the pseudo temporal features are output.

4. The audio information generating method according to claim 1, wherein: The step of re-convolving the compressed features using a convolution layer to obtain a re-convolution feature corresponding to at least one convolution kernel in the convolution layer includes: For any convolution kernel, convolution calculation is performed on the compressed feature and the convolution kernel to obtain a reconvolution feature corresponding to the convolution kernel; Traverse all convolution kernels to obtain the reconvolution features corresponding to each convolution kernel in the convolution layer.

5. The audio information generating method according to claim 1, wherein: Generating reference audio features according to all target audio features includes: According to the arrangement order of all convolution kernels, all target audio features searched based on the reconvolution features of the corresponding convolution kernels are spliced ​​to obtain a splicing result; The splicing result is determined as the reference audio feature.

6. The audio information generating method according to claim 1, wherein: After generating reference audio features according to all target audio features, the method further includes: Inputting the reference audio features into a trained discriminator and outputting a discrimination result; If the identification result is passed, the step of reconstructing the reference audio features and determining the feature reconstruction result as generated audio information corresponding to the image to be converted is performed.

7. The audio information generating method according to any one of claims 1 to 6, characterized in that: said reconstructing the reference audio features, Determining the feature reconstruction result as generated audio information corresponding to the image to be converted includes: Inputting the reference audio features into a trained melody decoder to reconstruct the melody and output first sub-audio information; Inputting the reference audio features into a trained denoising decoder to perform denoised audio reconstruction and output second sub-audio information; The average of the first sub-audio information and the second sub-audio information is used as the generated audio information.

8. An audio information generation device based on image analysis, characterized in that: The audio information generating device comprises: An image capture module is used to obtain an image to be converted and its image features, and to perform sliding extraction in the image to be converted using a preset sliding window according to a set step size to obtain N captured images, where N is an integer greater than zero; A first feature extraction module is used to input the N intercepted images into a trained temporal convolutional network model to extract temporal features and output pseudo temporal features corresponding to the image to be converted; A second feature extraction module is configured to input the pseudo time series features and the image features into a trained encoder for feature compression to obtain compressed features, and reconvolute the compressed features using a convolutional layer to obtain reconvolution features corresponding to at least one convolution kernel in the convolutional layer; A feature search module is configured to search, for any reconvolution feature, a preset audio feature set for an audio feature that is most similar to the reconvolution feature, and obtain a target audio feature corresponding to the reconvolution feature; The audio generation module is used to generate reference audio features based on all target audio features, perform feature reconstruction on the reference audio features, and determine the feature reconstruction results as generated audio information corresponding to the image to be converted.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the audio information generating method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the audio information generating method according to any one of claims 1 to 7 is implemented.

Images