Audio information generation method and device based on image analysis, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, in particular to an audio information generation method and device based on image analysis, equipment and a medium. According to the method, N intercepted images are extracted from an image to be converted in a sliding mode through a sliding window, the N intercepted images are input into a time sequence convolution network model, pseudo time sequence features are output, the pseudo time sequence features and image features are input into an encoder, compression features are obtained, the compression features are subjected to reconvolution, reconvolution features are obtained, and the image to be converted is obtained. Searching an audio feature most similar to the re-convolution feature in a preset audio feature set to obtain a target audio feature, generating a reference audio feature according to the target audio feature, performing feature reconstruction on the reference audio feature to obtain generated audio information, intercepting a to-be-converted image in a sliding mode, and obtaining the to-be-converted image. Therefore, the intercepted image contains richer image information, the feature representation capability is improved, the re-convolution feature is mapped into the target audio feature conforming to the audio habit, and the association degree of the image and the generated audio is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for generating audio information based on image analysis. Background Art

[0002] At present, in many application fields such as game voice-overs, short video soundtracks, and metaverse background sounds, image signals and audio signals have a strong correlation and can be combined to provide users with a better audio-visual experience. The existing image-based music matching method still mainly uses manual music matching, that is, providing image music through human composition.

[0003] However, the manual music arrangement method requires huge creation costs, and when using deep learning to generate audio, it is usually done by converting text to audio, such as the Noise2music model developed by Google. This method can generate audio that conforms to the preset theme. However, when applied to the task of converting images to audio, since the amount of information contained in the image is much greater than that in the text, it is difficult to describe the image information in detail only through the theme, resulting in a low degree of correlation between the image and the generated audio. Therefore, how to improve the correlation between the image and the generated audio has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device and medium for generating audio information based on image analysis to solve the problem of low correlation between images and generated audio.

[0005] In a first aspect, an embodiment of the present invention provides a method for generating audio information based on image analysis, the method for generating audio information comprising: Obtain an image to be converted and its image features, and perform sliding extraction in the image to be converted according to a set step size through a preset sliding window to obtain N intercepted images, where N is an integer greater than zero; Input the N intercepted images into a trained temporal convolutional network model to extract temporal features, and output pseudo temporal features corresponding to the images to be converted; Inputting the pseudo time series features and the image features into a trained encoder for feature compression to obtain compressed features, and re-convolving the compressed features using a convolution layer to obtain re-convolution features corresponding to at least one convolution kernel in the convolution layer; For any reconvolution feature, searching for an audio feature most similar to the reconvolution feature in a preset audio feature set to obtain a target audio feature corresponding to the reconvolution feature; A reference audio feature is generated according to all target audio features, feature reconstruction is performed on the reference audio feature, and a feature reconstruction result is determined as generated audio information corresponding to the image to be converted.

[0006] In a second aspect, an embodiment of the present invention provides an audio information generating device based on image analysis, the audio information generating device comprising: An image capture module is used to obtain an image to be converted and its image features, and to perform sliding extraction in the image to be converted according to a set step length through a preset sliding window to obtain N captured images, where N is an integer greater than zero; A first feature extraction module is used to input the N intercepted images into a trained temporal convolutional network model to extract temporal features, and output pseudo temporal features corresponding to the image to be converted; A second feature extraction module is used to input the pseudo time series feature and the image feature into a trained encoder for feature compression to obtain a compressed feature, and to re-convolute the compressed feature using a convolution layer to obtain a re-convolution feature corresponding to at least one convolution kernel in the convolution layer; A feature search module, configured to search for an audio feature most similar to the reconvolution feature in a preset audio feature set for any reconvolution feature, and obtain a target audio feature corresponding to the reconvolution feature; The audio generation module is used to generate reference audio features according to all target audio features, perform feature reconstruction on the reference audio features, and determine the feature reconstruction results as generated audio information corresponding to the image to be converted.

[0007] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the audio information generation method as described in the first aspect when executing the computer program.

[0008] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the audio information generating method as described in the first aspect is implemented.

[0009] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: The image to be converted and its image features are obtained, and sliding extraction is performed in the image to be converted according to a set step size through a preset sliding window to obtain N intercepted images, and the N intercepted images are input into a trained temporal convolutional network model to extract temporal features, and the pseudo temporal features corresponding to the image to be converted are output, and the pseudo temporal features and image features are input into a trained encoder for feature compression to obtain compressed features, and the compressed features are re-convolved using a convolutional layer to obtain a re-convolution feature corresponding to at least one convolution kernel in the convolutional layer, and for any re-convolution feature, the audio feature set that is most similar to the re-convolution feature is searched. Frequency features are obtained to obtain target audio features corresponding to the reconvolution features. Reference audio features are generated according to all target audio features. Feature reconstruction is performed on the reference audio features to determine the feature reconstruction results as generated audio information corresponding to the image to be converted. The image to be converted is intercepted by sliding so that the intercepted image contains different image information, thereby extracting pseudo-time series features. The reconvolution features can be mapped to target audio features that conform to audio habits through audio feature search, thereby generating audio, improving the coupling of image features and audio features, and thereby improving the correlation between the image and the generated audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0011] Figure 1 This is a schematic diagram of an application environment of an audio information generation method based on image analysis provided in Embodiment 1 of the present invention; Figure 2 It is a flowchart of a method for generating audio information based on image analysis provided in Embodiment 1 of the present invention; Figure 3 is a structural schematic diagram of an audio information generating device based on image analysis provided in Embodiment 2 of the present invention; Figure 4 It is a structural diagram of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0013] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0014] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0015] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0016] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0018] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0019] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0020] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0021] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0022] The first embodiment of the present invention provides an audio information generation method based on image analysis, which can be applied in the following aspects: Figure 1 In the application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, personal digital assistants (PDAs) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0023] See also Figure 2 , is a flow chart of a method for generating audio information based on image analysis provided by Embodiment 1 of the present invention. The method for generating audio information based on image analysis can be applied to Figure 1The server in the server, the computer device corresponding to the server communicates with the client to obtain the image to be converted and its image features provided by the client. The image to be converted can refer to the image that needs to generate audio information. The image features can be used to characterize the spatial information of the image to be converted. The computer device corresponding to the server is deployed with a trained temporal convolutional network model, a trained encoder and a convolutional layer. The trained temporal convolutional network model can be used to extract temporal features, the trained encoder can be used for feature compression, and the convolutional layer can be used for feature reconvolution processing. Figure 2 As shown, the audio information generation method based on image analysis may include the following steps: Step S201, obtaining an image to be converted and its image features, performing sliding extraction in the image to be converted by using a preset sliding window according to a set step length, and obtaining N captured images.

[0024] Among them, the image to be converted may refer to an image for which audio information generation is required, the image to be converted may be input into the client by a user, the image feature may represent the spatial feature information of the image to be converted, the preset sliding window may refer to a sliding window, the preset sliding window may be used to capture an image from the image to be converted, the set step size may refer to the sliding step size of the preset sliding window, the captured image may refer to an image captured in the process of the preset sliding window sliding on the image to be converted, and N is an integer greater than zero.

[0025] Specifically, the image to be converted needs to meet the preset size, and the preset size is set to W*H, where W is the width of the image to be converted and H is the height of the image to be converted. For example, the preset size can be set to 1024*648, and the size of the preset sliding window also meets the preset size, which is also W*H.

[0026] Optionally, performing sliding extraction in the image to be converted according to a set step size through a preset sliding window to obtain N intercepted images includes: Set the sliding direction, slide the preset sliding window according to the sliding direction and the set step size, retain all the pixel points in the intersection of the preset sliding window and the image to be converted after sliding, and set other pixel points to zero to obtain the intercepted image; Return to execute the step of sliding the preset sliding window according to the sliding direction and the set step size, retaining all pixel points in the intersection of the sliding preset sliding window and the image to be converted, and setting other pixel points to zero, to obtain the step of capturing the image, until the intersection of the sliding preset sliding window and the image to be converted is an empty set, stop iteration, and obtain N captured images.

[0027] The sliding direction may refer to a preset sliding direction of the sliding window.

[0028] Specifically, the initial position of the preset sliding window can be located on the left side of the image to be converted, so that the rightmost edge of the preset sliding window coincides with the leftmost edge of the image to be converted. At this time, the sliding direction is to the right, and the preset sliding window is slid according to the preset step size to extract N captured images.

[0029] The value of the set step size may be an integer greater than or equal to 1 and less than W. In the present embodiment, the value of the set step size is set to 16. For an image to be converted whose width W is 1024, the preset sliding window slides to the right 64 times so that the preset sliding window and the image to be converted completely overlap. Sliding to the right 128 times may make the leftmost edge of the preset sliding window overlap with the rightmost edge of the image to be converted. Thus, a total of 127 captured images may be obtained, i.e., N is 127.

[0030] The element values ​​of all elements in the preset sliding window can be set to 1. When the preset sliding window slides at any time, each element in the preset sliding window is multiplied by the element at the corresponding position in the image to be converted. If any element in the preset sliding window does not have an element at a corresponding position in the image to be converted, the element in the preset sliding window is multiplied by zero, and the preset sliding window after the element value multiplication is determined to be a captured image and a copy of the captured image is generated. The preset sliding window is initialized, that is, the element values ​​of all elements in the preset sliding window are set to 1, and the next sliding is performed until the intersection of the preset sliding window and the image to be converted after sliding is an empty set. At this time, each element in the preset sliding window does not have an element at a corresponding position in the image to be converted.

[0031] In one embodiment, the initial position of the preset sliding window can also be located at various positions such as the right side, top side, bottom side, and upper left side of the image to be converted. It is only necessary to set the sliding direction so that the preset sliding window can intersect with the image to be converted during the sliding process. For example, when the initial position of the preset sliding window is located on the right side of the image to be converted, the leftmost edge of the preset sliding window is made to coincide with the rightmost edge of the image to be converted. At this time, the sliding direction can be sliding to the left. The preset sliding window is slid according to a preset step size to extract N captured images.

[0032] In this embodiment, the captured images are extracted by sliding a preset sliding window, so that the image information contained in the captured images is different, so that when the feature extraction is performed subsequently, the model can extract different features according to different captured images. For example, for a landscape painting image, there is water at the bottom of the image and mountains at the top. When the preset sliding window slides from bottom to top, the first captured image extracted only includes water information, and then the captured image containing mountain and water information is extracted, and finally only the captured image containing mountain information is extracted. Each captured image contains different information, and the perception focus of the model is also different when extracting features, thereby effectively enriching the data input of the model. The above-mentioned steps of obtaining the image to be converted and its image features, performing sliding extraction in the image to be converted according to the set step size through a preset sliding window, and obtaining N captured images can make the subsequently extracted image features richer and more complete, thereby improving the accuracy of generating audio information based on the image, that is, improving the correlation between the image and the generated audio.

[0033] Step S202: input the N captured images into the trained temporal convolutional network model to extract temporal features, and output pseudo temporal features corresponding to the images to be converted.

[0034] Among them, the input of the trained temporal convolutional network model is data that conforms to the time sequence, and the output of the trained temporal convolutional network model is the temporal features. The pseudo-temporal features can be used to characterize the temporal features of N captured images.

[0035] Specifically, since images do not contain information in the time dimension, if you want to generate audio information containing timing information through images, you need to forge the image information into a set of images with timing information. The image set is N captured images, and the N captured images are extracted by sliding a preset window, which can approximately represent the process of gradually moving the field of view from one side of the image to the other side, and this process can be considered to be in a time sequence, that is, the N captured images can be approximately considered to be in a time sequence.

[0036] In one implementation, the implementer may directly use images of all frames in a continuous video segment as the N captured images. Under the constraint of the continuous video segment, the N captured images must be in time sequence.

[0037] In one embodiment, the implementer may first convert the image to be converted into data with time series information, such as a text description of the image, and then directly extract time series features from the text description through a trained time series convolutional network model, thereby also obtaining pseudo time series features corresponding to the image to be converted.

[0038] The trained temporal convolutional network model may include multiple hidden layers. The temporal convolutional network model uses a slider to determine a single input, and a single captured image is regarded as an input node. The size of the slider may include multiple input nodes. For example, the slider may include three input nodes. The movement of the slider also requires setting a corresponding step size, and the moving direction may be from left to right. In this embodiment, the step size may be set to 2, and the first input is [1, 2, 3]. The numbers therein may be used to identify the input node, that is, the position of the captured image in the sequence of N captured images. Since the step size is set to 2, the second input is [3, 4, 5], and so on.

[0039] The first hidden layer extracts the time series features of multiple captured images input each time to obtain the first sub-feature. When processing in the second hidden layer, the first sub-feature is used as the input node and slides with the same slider size, moving direction and step size. The second hidden layer extracts the time series features of multiple first sub-features input each time to obtain the second sub-feature.

[0040] As the number of hidden layers increases, each hidden layer uses the output of the previous hidden layer as an input node, and a slider is used to determine a single input until the number of hidden layers reaches a preset layer. At this time, only one input can be determined by the slider, and the time series features of the input are extracted to obtain the final integrated features, which are the pseudo-time series features of the N captured images.

[0041] It should be noted that, in this embodiment, since the final output is a pseudo-time series feature, only the encoder part of the trained time convolutional network model is needed. In order to ensure the effectiveness of the encoder part in extracting time series features, during the training process of the time convolutional network model, a classifier can be connected behind the encoder part, and the training samples are input into the encoder part for feature extraction to obtain predicted time series features, and then the predicted time series features are input into the classifier, and the predicted classification results are output. The loss function is calculated according to the predicted classification results and the labels of the corresponding training samples to obtain the training loss. The parameters of the encoder part and the classifier are adjusted according to the training loss until the training loss converges to obtain the trained encoder part. The loss function can use a cross entropy loss function, and the label can refer to an emotional label. The emotional label can represent the image of the image, and can also represent the emotional information of the audio information, which is suitable as a label in this embodiment. The emotional label can include a happy label, a sad label, a fear label, etc.

[0042] Optionally, N intercepted images are input into a trained temporal convolutional network model to extract temporal features, and the pseudo temporal features of the output corresponding to the image to be converted include: According to the extraction order, an image sequence is formed by N intercepted images; The image sequence is input into the trained temporal convolutional network model to extract temporal features and output pseudo-temporal features.

[0043] The extraction order may refer to the order in which the captured images are captured, and the image sequence may include N captured images that conform to the extraction order.

[0044] Specifically, the extraction order can be approximately regarded as the time order, so that the N captured images contained in the image sequence approximately conform to the time order, so that the time series features can be extracted through the trained time series convolutional network model. At this time, the extracted features are pseudo-time features.

[0045] The above-mentioned step of inputting N captured images into the trained temporal convolutional network model for temporal feature extraction and outputting the pseudo-temporal features corresponding to the image to be converted can effectively extract the temporal features of the captured images, thereby facilitating the provision of temporal information for audio information generation and increasing the degree of correlation between the image and the generated audio information.

[0046] Step S203, input the pseudo time series features and the image features into the trained encoder for feature compression to obtain compressed features, and use the convolution layer to re-convolute the compressed features to obtain re-convolution features corresponding to at least one convolution kernel in the convolution layer.

[0047] Among them, the trained encoder can be used to further compress the pseudo-time series features and image features. The compressed features can represent the spatiotemporal features of the image to be converted. The convolution layer can be used to further extract the compressed features, that is, re-convolution. The convolution layer can include at least one convolution kernel, and one convolution kernel corresponds to a convolution result, which is also the re-convolution feature.

[0048] Specifically, the trained encoder may include two encoding branches, the input of one encoding branch is the pseudo-time series feature, and the input of the other encoding branch is the image feature. The purpose of setting the two encoding branches is to solve the problem of inconsistent sizes of the pseudo-time series features and the image features, to constrain the output sizes of the two encoding branches to be consistent, and to add the encoding features obtained by the two encoding branches point by point, and to determine that the point-by-point addition result is a compressed feature.

[0049] Since the compressed features are used to characterize the spatiotemporal features of the image to be converted, the spatiotemporal features should be correlated with the audio information. Therefore, in the training process of the encoder, an additional encoding branch of the audio information can be added, and the encoding branch of the audio information can be trained. Specifically, the sample acquisition in the training process can be extracted from scenes such as game scenes and short video scenes with soundtracks. For a training image sample, there is an associated audio information sample. The training image sample is input into the first encoding branch and the second encoding branch to obtain image sample features and pseudo-time sample features. The audio information sample is input into the third encoding branch to obtain audio sample features. The image sample features and the pseudo-time sample features are weighted added for the first time to obtain compressed sample features. The weights of the first weighted addition can be averaged. It is set to 1, and the implementer can adjust it according to the actual situation. Then, the compressed sample features and the audio sample features are weighted added for the second time. In this embodiment, the weights of the second weighted addition can be set to 0.5. The implementer can adjust it according to the actual situation. The result of the second weighted addition is input into the temporary classifier, and the reference classification result is output. According to the labels corresponding to the training image samples and their associated audio information samples and the reference classification results, the first encoding branch and the second encoding branch are trained to obtain the trained encoder. The label can still be an emotion label. In the training process of the encoder, the association between audio information, image features and pseudo-time series features is directly modeled. Compared with the training of the above-mentioned time convolutional network model, the correlation between image information and audio information is further enhanced.

[0050] Optionally, the compressed features are reconvolved using a convolution layer, and the reconvolution features corresponding to at least one convolution kernel in the convolution layer include: For any convolution kernel, convolution is performed on the compressed feature and the convolution kernel to obtain the reconvolution feature corresponding to the convolution kernel; Traverse all convolution kernels and obtain the reconvolution features corresponding to each convolution kernel in the convolution layer.

[0051] Among them, the convolution kernel can be used for convolution operation. In this embodiment, the compressed feature can represent the spatiotemporal information. The number of convolution kernels is set to be consistent with the time dimension of the compressed feature. At this time, the reconvolution feature corresponding to each convolution kernel can represent the deep features of each moment.

[0052] Specifically, the training of the convolution layer can be supervised by real audio features, and the training samples can be compressed sample features. The audio samples associated with the images corresponding to the compressed sample features are obtained. The audio samples are framed according to the number of convolution kernels to obtain the same number of frame results as the number of convolution kernels. The frame features corresponding to each frame result are extracted, and the frame features can be used as the real audio features. The similarity between the re-convolution features and the frame features is calculated as the loss function for the convolution layer training. The convolution layer is trained to obtain a trained convolution layer.

[0053] Step S204: for any reconvolution feature, search for an audio feature that is most similar to the reconvolution feature in a preset audio feature set to obtain a target audio feature corresponding to the reconvolution feature.

[0054] Among them, the preset audio feature set may include several audio features, the audio feature may refer to the feature corresponding to the audio information that conforms to the audio habits, the audio information that conforms to the audio habits may be chords, etc., and the target audio feature may refer to the audio feature that is closest to the reconvolution feature.

[0055] Specifically, since the present embodiment needs to convert the image into audio information, the conventional method of extracting image features for reconstruction will inevitably result in a large amount of noise, resulting in poor audio information generation. Therefore, the present embodiment presets an audio feature set. Before feature reconstruction, the reconvolution feature needs to be searched in the audio feature set to obtain the audio feature that is most similar to the reconvolution feature, that is, the target audio feature. The target audio feature can not only ensure the retention of image features, but also ensure that the audio features used for reconstruction conform to audio information habits.

[0056] The similarity between the audio feature and the reconvolution feature can be measured by cosine similarity, Euclidean distance, etc. In this embodiment, cosine similarity is used to calculate the similarity, and the value range of the calculation result is [0, 1]. The closer the calculation result is to 1, the greater the similarity between the audio feature and the reconvolution feature. The closer the calculation result is to 0, the smaller the similarity between the audio feature and the reconvolution feature. For any reconvolution feature, the similarity between the reconvolution feature and each audio feature in the audio feature set is calculated, and the audio feature corresponding to the maximum similarity is determined as the target audio feature of the corresponding reconvolution feature.

[0057] The above-mentioned step of searching for the audio feature most similar to the reconvolution feature in the preset audio feature set for any reconvolution feature to obtain the target audio feature corresponding to the reconvolution feature, and obtaining the target audio feature corresponding to the reconvolution feature by searching, can not only ensure the retention of image features, but also ensure that the audio features used for reconstruction conform to the audio information habits, thereby improving the accuracy of subsequent audio information generation.

[0058] Step S205: Generate reference audio features based on all target audio features, perform feature reconstruction on the reference audio features, and determine the feature reconstruction results as generated audio information corresponding to the image to be converted.

[0059] The reference audio features may include all target audio features, the reference audio features may be used for reconstructing audio information, and the generated audio information may refer to the generated result of the audio information.

[0060] Specifically, feature reconstruction can be achieved using a trained decoder, and the training process of the decoder can adopt a pre-training method, that is, using audio feature samples to form a training data set, and the audio information of the corresponding audio feature samples as labels, the decoder is trained based on the generated sample audio information and labels obtained by inputting the audio feature samples into the decoder to obtain a trained decoder.

[0061] Optionally, generating reference audio features according to all target audio features includes: According to the arrangement order of all convolution kernels, all target audio features searched according to the reconvolution features of the corresponding convolution kernels are spliced ​​to obtain a splicing result; The splicing result is determined as the reference audio feature.

[0062] Among them, the arrangement order of all convolution kernels can refer to the arrangement order of the corresponding time dimension, the splicing can be performed in a concatenated manner, and the splicing result can refer to the splicing result of all reconvolution features. Since the generation of audio information usually requires the generation of multiple frames of audio rather than a single frame of audio, it is necessary to determine the corresponding target audio features for each frame of audio.

[0063] Optionally, after generating reference audio features according to all target audio features, the method further includes: Input the reference audio features into the trained discriminator and output the identification result; If the identification result is passed, the step of reconstructing the reference audio features and determining the feature reconstruction result as the generated audio information corresponding to the image to be converted is executed.

[0064] Among them, the trained discriminator can be used to identify whether the reference audio feature is a noise feature. The identification result can include pass and fail. Pass can indicate that the reference audio feature is not a noise feature, and fail can indicate that the reference audio feature is a noise feature.

[0065] Specifically, the training of the discriminator can use noisy audio features and non-noise audio features as training data, and the training data is labeled, and the labeling categories are also passed and failed. It should be noted that in the labeling process, non-noise audio features do not mean that there is no noise at all. As long as the audio information corresponding to the training data can conform to normal audio habits, it is considered to be a non-noise audio feature. For example, the audio information corresponding to the training data is a melody, but contains slight white noise. At this time, the training data is still considered to be labeled as a non-noise audio feature.

[0066] If the identification result is passed, it means that the reference audio feature contains valid audio information at this time, and the step of reconstructing the reference audio feature and determining that the feature reconstruction result is the generated audio information corresponding to the image to be converted is executed.

[0067] Optionally, reconstructing the reference audio features and determining the feature reconstruction result as generated audio information corresponding to the image to be converted includes: Input the reference audio feature into the trained melody decoder to reconstruct the melody and output the first sub-audio information; Input the reference audio features into the trained denoising decoder to reconstruct the denoised audio and output the second sub-audio information; The average value of the first sub audio information and the second sub audio information is used as the generated audio information.

[0068] Among them, the trained melody decoder is used for melody reconstruction, that is, it pays more attention to the melody and time correlation of the generated audio information, but ignores the quality of the generated audio information, that is, the audio information generated by the trained melody decoder may contain certain noise, and the first sub-audio information can refer to the result of melody reconstruction in the trained melody decoder input with reference to audio features.

[0069] The trained denoising decoder is used for denoised audio reconstruction, that is, it pays more attention to the quality of the generated audio information, but ignores the melody of the generated audio information, that is, the audio information generated by the trained denoising decoder may not have melody, and the first sub-audio information may refer to the result of denoised audio reconstruction by inputting the reference audio features into the trained denoising decoder.

[0070] Specifically, mean calculation is essentially a special case of weighted calculation. At this time, the weights of the weighted calculation of the first sub-audio information and the second sub-audio information are both 0.5. The implementer can adjust the weights according to the actual needs of generating audio information. For example, if more attention is paid to the melody part, the weight of the first sub-audio information can be increased, and the weight of the second sub-audio information can be reduced. They can be set to 0.6 and 0.4 respectively.

[0071] The above steps of generating reference audio features based on all target audio features, reconstructing the reference audio features, and determining the feature reconstruction results as the generated audio information corresponding to the image to be converted. Since the target audio features are used to form the reference audio features, the generated audio information reconstructed based on the reference audio features is more in line with audio habits, thereby improving the audio quality and the correlation between the generated audio and the image.

[0072] In this embodiment, through audio feature search, the reconvolution features can be mapped to target audio features that conform to audio habits, thereby generating audio, improving the coupling between image features and audio features, and further improving the correlation between the image and the generated audio.

[0073] Corresponding to the audio information generation method based on image analysis of the above embodiment, Figure 3 The structural block diagram of the audio information generation device based on image analysis provided in the second embodiment of the present invention is shown. The above-mentioned audio information generation device based on image analysis is applied to the server. The computer device corresponding to the server communicates with the client to obtain the image to be converted and its image features provided by the client. The image to be converted may refer to the image for which audio information generation is required. The image features can be used to characterize the spatial domain information of the image to be converted. The computer device corresponding to the server is deployed with a trained temporal convolutional network model, a trained encoder and a convolutional layer. The trained temporal convolutional network model can be used to extract temporal features, the trained encoder can be used to perform feature compression, and the convolutional layer can be used for feature reconvolution processing. For the convenience of explanation, only the parts related to the embodiment of the present invention are shown.

[0074] See also Figure 3 , the audio information generating device based on image analysis comprises: The image capture module 31 is used to obtain the image to be converted and its image features, and to perform sliding extraction in the image to be converted according to a set step size through a preset sliding window to obtain N captured images, where N is an integer greater than zero; The first feature extraction module 32 is used to input the N intercepted images into the trained temporal convolutional network model to extract temporal features, and output pseudo temporal features corresponding to the image to be converted; A second feature extraction module 33 is used to input the pseudo time series features and the image features into the trained encoder for feature compression to obtain compressed features, and to re-convolute the compressed features using a convolution layer to obtain re-convolution features corresponding to at least one convolution kernel in the convolution layer; A feature search module 34 is used to search for an audio feature that is most similar to the reconvolution feature in a preset audio feature set for any reconvolution feature, and obtain a target audio feature corresponding to the reconvolution feature; The audio generation module 35 is used to generate reference audio features according to all target audio features, perform feature reconstruction on the reference audio features, and determine the feature reconstruction results as generated audio information corresponding to the image to be converted.

[0075] Optionally, the image capture module 31 includes: A sliding window sliding unit is used to set a sliding direction, slide a preset sliding window according to the sliding direction and the set step length, retain all pixel points in the intersection of the preset sliding window and the image to be converted after sliding, and set other pixel points to zero to obtain a captured image; The iterative sliding unit is used to return to the step of sliding the preset sliding window according to the sliding direction and the set step size, retaining all pixel points in the intersection of the sliding preset sliding window and the image to be converted, and setting other pixel points to zero, to obtain the intercepted image, until the intersection of the sliding preset sliding window and the image to be converted is an empty set, stop the iteration, and obtain N intercepted images.

[0076] Optionally, the first feature extraction module 32 includes: A sequence forming unit, used for forming an image sequence from the N captured images according to the extraction order; The temporal feature extraction unit is used to input the image sequence into the trained temporal convolutional network model to extract the temporal features and output pseudo temporal features.

[0077] Optionally, the second feature extraction module 33 includes: A convolution calculation unit, used for performing convolution calculation on the compressed feature and the convolution kernel for any convolution kernel, to obtain a reconvolution feature corresponding to the convolution kernel; The convolution kernel traversal unit is used to traverse all convolution kernels and obtain the reconvolution features corresponding to each convolution kernel in the convolution layer.

[0078] Optionally, the audio generation module 35 includes: A feature concatenation unit is used to concatenate all target audio features searched according to the reconvolution features of the corresponding convolution kernels according to the arrangement order of all convolution kernels to obtain a concatenation result; The reference feature determination unit is used to determine the splicing result as a reference audio feature.

[0079] Optionally, the above-mentioned audio information generating device further includes: A feature identification module, used to input reference audio features into the trained discriminator and output identification results; The step execution module is used to execute the step of reconstructing the reference audio features and determining the feature reconstruction result as the generated audio information corresponding to the image to be converted if the identification result is passed.

[0080] Optionally, the audio generation module 35 includes: A first reconstruction unit, used for inputting the reference audio feature into a trained melody decoder to reconstruct the melody and outputting first sub-audio information; The reconstruction unit is used to input the reference audio feature into the trained denoising decoder to perform denoised audio reconstruction and output the second sub-audio information; The mean value calculation unit is used to generate audio information using the mean value of the first sub audio information and the second sub audio information.

[0081] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules and units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0082] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned embodiments of the method for generating audio information based on image analysis are implemented.

[0083] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0084] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0085] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0086] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0087] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.

[0088] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0090] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0091] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for generating audio information based on image analysis, characterized in that: The audio information generation method comprises: Obtain an image to be converted and its image features, and perform sliding extraction in the image to be converted according to a set step size through a preset sliding window to obtain N intercepted images, where N is an integer greater than zero; Input the N intercepted images into a trained temporal convolutional network model to extract temporal features, and output pseudo temporal features corresponding to the images to be converted; Inputting the pseudo time series features and the image features into a trained encoder for feature compression to obtain compressed features, and re-convolving the compressed features using a convolution layer to obtain re-convolution features corresponding to at least one convolution kernel in the convolution layer; For any reconvolution feature, searching for an audio feature most similar to the reconvolution feature in a preset audio feature set to obtain a target audio feature corresponding to the reconvolution feature; A reference audio feature is generated according to all target audio features, feature reconstruction is performed on the reference audio feature, and a feature reconstruction result is determined as generated audio information corresponding to the image to be converted.

2. The audio information generating method according to claim 1, characterized in that: The step of performing sliding extraction in the image to be converted by using a preset sliding window according to a set step length to obtain N intercepted images includes: Setting a sliding direction, sliding the preset sliding window according to the sliding direction and the set step size, retaining all pixel points in the intersection of the preset sliding window and the image to be converted after sliding, and setting other pixel points to zero, to obtain the intercepted image; Return to the step of sliding the preset sliding window according to the sliding direction and the set step size, retaining all pixel points in the intersection of the sliding preset sliding window and the image to be converted, and setting other pixel points to zero, to obtain the captured image, until the intersection of the sliding preset sliding window and the image to be converted is an empty set, stop iteration, and obtain the N captured images.

3. The audio information generating method according to claim 1, characterized in that: The step of inputting the N intercepted images into a trained temporal convolutional network model to extract temporal features and outputting pseudo temporal features corresponding to the image to be converted includes: According to the extraction order, the N captured images form an image sequence; The image sequence is input into the trained temporal convolutional network model to extract temporal features, and the pseudo temporal features are output.

4. The audio information generating method according to claim 1, characterized in that: The step of re-convolving the compressed features using a convolution layer to obtain a re-convolution feature corresponding to at least one convolution kernel in the convolution layer includes: For any convolution kernel, convolution calculation is performed on the compressed feature and the convolution kernel to obtain a reconvolution feature corresponding to the convolution kernel; Traverse all convolution kernels to obtain the reconvolution features corresponding to each convolution kernel in the convolution layer.

5. The audio information generating method according to claim 1, characterized in that: The generating of reference audio features according to all target audio features comprises: According to the arrangement order of all convolution kernels, all target audio features searched according to the reconvolution features of the corresponding convolution kernels are spliced ​​to obtain a splicing result; The splicing result is determined to be the reference audio feature.

6. The audio information generating method according to claim 1, characterized in that: After generating reference audio features according to all target audio features, the method further includes: Inputting the reference audio features into a trained discriminator and outputting a discrimination result; If the identification result is passed, the step of reconstructing the reference audio features and determining the feature reconstruction result as generated audio information corresponding to the image to be converted is performed.

7. The audio information generating method according to any one of claims 1 to 6, characterized in that: The feature reconstruction of the reference audio feature is performed, Determining the feature reconstruction result as generated audio information corresponding to the image to be converted includes: Inputting the reference audio feature into a trained melody decoder to reconstruct the melody and output first sub-audio information; Input the reference audio features into a trained denoising decoder to perform denoised audio reconstruction, and output second sub-audio information; The average of the first sub audio information and the second sub audio information is used as the generated audio information.

8. An audio information generating device based on image analysis, characterized in that: The audio information generating device comprises: An image capture module is used to obtain an image to be converted and its image features, and to perform sliding extraction in the image to be converted according to a set step size through a preset sliding window to obtain N captured images, where N is an integer greater than zero; A first feature extraction module is used to input the N intercepted images into a trained temporal convolutional network model to extract temporal features, and output pseudo temporal features corresponding to the image to be converted; A second feature extraction module is used to input the pseudo time series feature and the image feature into a trained encoder for feature compression to obtain a compressed feature, and to reconvolute the compressed feature using a convolution layer to obtain a reconvolution feature corresponding to at least one convolution kernel in the convolution layer; A feature search module, configured to search for an audio feature most similar to the reconvolution feature in a preset audio feature set for any reconvolution feature, and obtain a target audio feature corresponding to the reconvolution feature; The audio generation module is used to generate reference audio features according to all target audio features, perform feature reconstruction on the reference audio features, and determine the feature reconstruction results as generated audio information corresponding to the image to be converted.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the audio information generating method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the audio information generating method according to any one of claims 1 to 7 is implemented.

Images