Audio storage method, device, electronic device and computer readable medium

Abstract

The embodiments of the present disclosure disclose an audio storage method, device, electronic device and computer-readable medium. A specific implementation of the method includes: performing video frame extraction processing on the target video to generate a frame image sequence; extracting subtitle information to obtain a subtitle information sequence; for each adjacent two subtitle information, executing the first generation step: determining a frame image subsequence; generating segment description information; adding object statement information to each subtitle information to generate added subtitle information and obtain an added subtitle information sequence; fusing the segment description information sequence and the added subtitle information sequence to generate a fused information sequence; generating a video explanation text; extracting a first audio sequence; generating a second audio sequence; generating an explanation audio; and correspondingly storing the explanation audio and the video identifier corresponding to the target video. This implementation can accurately and efficiently generate the explanation audio, so that the subsequent target user can perform an audio explanation for the explanation audio.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to an audio storage method, device, electronic device, and computer-readable medium. Background Art

[0002] At present, applications based on video to audio conversion have been widely used in people's lives. For video to audio explanation, the usual method is: first, extract the audio from the video. Then, add the narration corresponding to the audio to generate the audio explanation.

[0003] However, when using the above method to generate explanation audio, the following technical problems often occur:

[0004] The generated audio explanation has limited explanation content and is incoherent, resulting in poor audio explanation effect.

[0005] In the process of adopting technical solutions to solve the above-mentioned technical problem 1, the following technical problem is often accompanied: how to generate accurate explanation audio for the first audio sequence corresponding to the subtitle information sequence and the second audio sequence of the segment description information sequence. In response to the above-mentioned technical problem, the conventional solution is generally: directly combine the first audio sequence and the second audio sequence to generate a combined audio as the explanation audio. However, the above-mentioned conventional solution still has the following problem 2: the audio content of the obtained explanation audio may be not smooth, and explanation errors may occur.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0007] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0008] Some embodiments of the present disclosure propose an audio storage method, an apparatus, an electronic device, and a computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0009] In a first aspect, some embodiments of the present disclosure provide an audio storage method, comprising: performing video frame extraction processing on a target video to generate a frame image sequence; extracting subtitle information corresponding to the target video to obtain a subtitle information sequence; for each adjacent two subtitle information in the subtitle information sequence, performing the following first generation step: determining a frame image subsequence in the frame image sequence between the two adjacent subtitle information; generating segment description information for the frame image subsequence; adding corresponding object statement information to each subtitle information in the subtitle information sequence to generate added subtitle information to obtain an added subtitle information sequence; according to a time sequence relationship, performing corresponding information fusion on the obtained segment description information sequence and the added subtitle information sequence to generate a fused information sequence; generating a video explanation text for the fused information sequence; extracting a first audio sequence corresponding to the subtitle information sequence in the target video; generating a second audio sequence for the segment description information sequence; generating an explanation audio for the video explanation text based on the first audio sequence and the second audio sequence; and storing the explanation audio and the video identifier corresponding to the target video in a corresponding manner.

[0010] In a second aspect, some embodiments of the present disclosure provide an audio storage device, comprising: a video frame extraction processing unit, configured to perform video frame extraction processing on a target video to generate a frame image sequence; a first extraction unit, configured to extract subtitle information corresponding to the target video to obtain a subtitle information sequence; an execution unit, configured to perform the following first generation step for each two adjacent subtitle information in the subtitle information sequence: determining a frame image subsequence in the frame image sequence between the two adjacent subtitle information; generating fragment description information for the frame image subsequence; an adding unit, configured to add corresponding object statement information to each subtitle information in the subtitle information sequence to generate added subtitle information, and obtain an added subtitle information sequence. ; The information fusion unit is configured to fuse the obtained segment description information sequence and the above-mentioned added subtitle information sequence according to the time sequence to generate a fused information sequence; the first generation unit is configured to generate a video explanation text for the above-mentioned fused information sequence; the second extraction unit is configured to extract the first audio sequence corresponding to the above-mentioned subtitle information sequence in the above-mentioned target video; the second generation unit is configured to generate a second audio sequence for the above-mentioned segment description information sequence; the third generation unit is configured to generate an explanation audio for the above-mentioned video explanation text based on the above-mentioned first audio sequence and the above-mentioned second audio sequence; the storage unit is configured to store the above-mentioned explanation audio and the video identifier corresponding to the above-mentioned target video in a corresponding manner.

[0011] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0012] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0013] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the audio storage method of some embodiments of the present disclosure can accurately and efficiently generate the explanation audio, so that the target user can perform the audio explanation for the explanation audio later. Specifically, the reason why the relevant explanation audio is not accurate is that the explanation content of the generated explanation audio is limited and the explanation content is incoherent, resulting in poor audio explanation effect. Based on this, the audio storage method of some embodiments of the present disclosure first performs video frame extraction processing on the target video to generate a frame image sequence, so as to facilitate the subsequent extraction of the corresponding subtitle information and the generation of the corresponding text content of the explanation audio. Then, the subtitle information corresponding to the above target video is extracted to obtain the subtitle information sequence, which is used as an important text feature in the subsequent explanation audio, effectively ensuring the correctness of the key content of the subsequent explanation audio. Then, for each adjacent two subtitle information in the above subtitle information sequence, the following first generation step is performed: the first step is to determine the frame image subsequence in the above frame image sequence between the above two adjacent subtitle information, so as to facilitate the subsequent generation of segment description information without subtitle information. The second step is to generate segment description information for the above frame image subsequence to enrich the audio content of the subsequent explanation audio. Next, corresponding object statement information is added to each subtitle information in the above subtitle information sequence to generate added subtitle information and obtain an added subtitle information sequence, so that subsequent listeners can clearly understand the explanation object of the subtitle and the substantive content of the corresponding subtitle information. Further, according to the time sequence, the obtained segment description information sequence and the above added subtitle information sequence are fused with corresponding information to generate a fused information sequence to obtain a preliminary and perfect explanation text. On this basis, a video explanation text for the above fused information sequence can be accurately generated to further enhance the expressiveness and richness of the content on the basis of the fused information sequence. Secondly, the first audio sequence corresponding to the above subtitle information sequence in the above target video is extracted to obtain the corresponding audio feature content. In addition, a second audio sequence for the above segment description information sequence can be accurately generated. Furthermore, according to the above first audio sequence and the above second audio sequence, an explanation audio with not only correct but also rich content can be accurately generated for the above video explanation text. Finally, the above explanation audio and the video identifier corresponding to the above target video are stored correspondingly, so that the subsequent target user can perform audio explanation for the explanation audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0015] Figure 1 is a flow chart of some embodiments of the audio storage method according to the present disclosure;

[0016] Figure 2 is a schematic structural diagram of some embodiments of the audio storage device according to the present disclosure;

[0017] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] refer to Figure 1 , shows a process 100 of some embodiments of the audio storage method according to the present disclosure. The audio storage method comprises the following steps:

[0025] Step 101: Perform frame extraction processing on the target video to generate a frame image sequence.

[0026] In some embodiments, the execution subject of the above audio storage method can perform video frame extraction processing on the target video to generate a frame image sequence. The target video can be a video to be converted into an explanation audio. The explanation audio is an audio that can be used to explain the video content in the form of voice playback. In practice, the target video can be a target skit.

[0027] As an example, the execution subject may perform frame extraction processing on the target video according to a preset frame extraction duration to generate a frame image sequence.

[0028] Step 102: extract the subtitle information corresponding to the target video to obtain a subtitle information sequence.

[0029] In some embodiments, the execution subject may extract the subtitle information corresponding to the target video to obtain a subtitle information sequence, wherein the subtitle information sequence may be each video subtitle in the target video.

[0030] As an example, the execution subject may input each frame image in the frame image sequence into a text recognition model to generate text information as subtitle information, thereby obtaining a subtitle information sequence.

[0031] Step 103: for each two adjacent subtitle information in the subtitle information sequence, perform the following first generation step:

[0032] Step 1031: determine a frame image subsequence in the frame image sequence between the two adjacent subtitle information.

[0033] In some embodiments, the execution entity may determine a frame image subsequence in the frame image sequence between the two adjacent subtitle information.

[0034] As an example, first, the execution subject may determine the frame image corresponding to each of the two adjacent subtitle information as the target frame image, and obtain the target frame image group. Then, the image time corresponding to each target frame image in the target frame image group is determined to obtain the image time group. Finally, at least one frame image in the frame image sequence that is in the image time group is determined as the frame image subsequence.

[0035] Step 1032: Generate segment description information for the above frame image subsequence.

[0036] In some embodiments, the execution subject may generate segment description information for the frame image subsequence, wherein the segment description information may be information that describes the video segment corresponding to the frame image subsequence in text.

[0037] In some optional implementations of some embodiments, the generating of the segment description information for the frame image subsequence may include the following steps:

[0038] In the first step, each frame image in the above frame image subsequence is input into a pre-trained image feature extraction model to generate image feature information and obtain an image feature information subsequence. The image feature extraction model can be a neural network model for extracting image features. In practice, the image feature extraction model can be a convolutional neural network model with multiple layers connected in series. The image feature information can characterize the image feature semantics of the frame image. The image feature extraction model can be a neural network model trained based on the gradient descent method. The input of the image feature extraction model is an image, and the output is the feature information corresponding to the image.

[0039] The second step is to perform clustering processing on each image feature information in the above image feature information subsequence to generate an image feature information cluster set, wherein the time corresponding to each image feature information in the image feature information cluster is continuous.

[0040] As an example, the execution entity may utilize a k-means algorithm to perform clustering processing on each image feature information in the image feature information subsequence to generate an image feature information cluster.

[0041] The third step is to adjust the model structure of the acquired image set partition information model according to the number of clusters to generate an adjusted image set partition information model. The number of clusters may be the number of clusters of each image feature information cluster included in the image feature information cluster set.

[0042] The fourth step is to generate an image feature information group set for the image feature information subsequence using the adjusted image set division information model, wherein each image feature information in each image feature information group in the image feature information group set may be feature information with high feature information similarity.

[0043] The fifth step is to divide the frame image subsequence according to the image feature information cluster set and the image feature information group set to generate a divided frame image group sequence.

[0044] As an example, first, the execution subject may determine the frame image cluster corresponding to the image feature information cluster, and determine the frame image group corresponding to the image feature information group. Then, the image division difference information between the frame image cluster and the frame image group is determined. Finally, in response to determining that the image division difference information represents that there is no difference, the frame image group sequence corresponding to the frame image group is determined as the divided frame image group sequence. In response to determining that the image division difference information represents that there is a difference, at least one difference image is determined. Next, for each difference image in the at least one difference image, first, the similarity between the difference image and each frame image group is determined. Then, the difference image is classified into the frame image group with the highest corresponding similarity. The frame image group sequence corresponding to the obtained divided frame image group is determined as the divided frame image group sequence.

[0045] Step 6: Generate segment description information based on the above-mentioned divided frame image group sequence.

[0046] As an example, first, the execution subject may generate image group description information corresponding to each divided frame image group to obtain an image group description information sequence, and then determine the image group description information sequence as the segment description information.

[0047] In some optional implementations of some embodiments, the image set partitioning information model includes: an image set partitioning model and multiple image similarity determination models, wherein the number of image similarity determination models included in the multiple image similarity determination models is a predetermined number. The predetermined number is greater than the number of clusters. The number of image sets output by the image set partitioning model is the same as the predetermined number. The image set partitioning model may be a model for performing image partitioning on an image set. In practice, the image set partitioning model may include: multiple partitioning parameters. Each partitioning parameter may be an integer. The numerical value corresponding to each partitioning parameter represents how to perform image partitioning on the input image set. The image similarity determination model may be a neural network model for generating image similarities between images in the divided image subsets. The image similarity may be a numerical value between 0 and 1. The larger the numerical value, the more similar the images in the corresponding divided image subsets are. In practice, the image similarity determination model may be a convolutional neural network model.

[0048] Optionally, the step of adjusting the model structure of the acquired image set partition information model according to the number of clusters to generate an adjusted image set partition information model may include the following steps:

[0049] In the first step, the initial partitioning parameters included in the image set partitioning model are trimmed according to the number of clusters to generate a trimmed image set partitioning model, wherein the number of parameters of the partitioning parameters included in the trimmed image set partitioning model is the number of clusters.

[0050] The second step is to perform model cropping on each of the plurality of image similarity determination models according to the outputs of each model corresponding to the cropped image set partitioning model, so as to obtain a plurality of cropped image similarity determination models. The input of each model in the plurality of cropped image similarity determination models corresponds to the outputs of each model corresponding to the cropped image set partitioning model.

[0051] In some optional implementations of some embodiments, the above-mentioned use of the adjusted image set partitioning information model to generate the image feature information group set for the above-mentioned image feature information subsequence may include the following steps:

[0052] The first step is to perform the following second generation step for the above image feature information subsequence:

[0053] Sub-step 1: using the above-mentioned image set division model, divide the above-mentioned image feature information subsequence into feature information to generate an initial image feature information group set.

[0054] Sub-step 2, inputting each initial image feature information group in the initial image feature information group set into the above-mentioned cropped multiple image similarity determination models to generate an image similarity set. There is a one-to-one correspondence between the image similarities in the image similarity set and the initial image feature information groups in the initial image feature information group set. The image similarity can represent the similarity between each initial image feature information in the initial image feature information group.

[0055] Sub-step 3: in response to determining that the values ​​corresponding to the respective image similarities in the image similarity set are greater than a preset value, the initial image feature information set is determined as the image feature information set. For example, the preset value may be 0.87.

[0056] In the second step, in response to determining that there are image similarities with corresponding values ​​less than the preset value in the image similarity set, at least one image similarity with a value less than the preset value is determined.

[0057] The third step is to determine the difference between each image similarity in the at least one image similarity and the preset value to generate a difference value, thereby obtaining at least one difference value.

[0058] The fourth step is to adjust at least one initial partitioning parameter included in the image set partitioning model according to the at least one phase difference value to obtain an adjusted image set partitioning model.

[0059] As an example, the execution entity may utilize an association table between difference values ​​and parameter adjustment sizes to adjust at least one initial partitioning parameter included in the image set partitioning model according to the at least one phase difference value to obtain an adjusted image set partitioning model.

[0060] The fifth step is to use the adjusted image set partition model as the image set partition model and continue to execute the second generation step.

[0061] In some optional implementations of some embodiments, the generation of the segment description information according to the division of the frame image group sequence may include the following steps:

[0062] In the first step, for the target divided frame image group in the divided frame image group sequence, the following third generation step is performed:

[0063] Sub-step 1: determine the image feature information corresponding to each divided frame image in the target divided frame image group as the target image feature information to obtain the target image feature information group, wherein the image feature information can represent the image feature semantics corresponding to the divided frame image.

[0064] Sub-step 2, input the above target image feature information group into the feature fusion model to generate fused feature information. The feature fusion model can be a pre-trained neural network model for information fusion of each input feature information. In practice, the feature fusion model can be a multi-head attention mechanism model based on a convolutional neural network model. The fused feature information can represent the fused features corresponding to each target image feature information in the target image feature information group.

[0065] Sub-step 3: sorting the feature information of each target image feature information in the target image feature information group to generate an image feature information sequence.

[0066] As an example, the execution subject may sort the feature information of each target image feature information in the target image feature information group according to the time sequence of the corresponding images to generate an image feature information sequence.

[0067] Sub-step 4: for the target image feature information in the image feature information sequence, perform the following fourth generation step:

[0068] The first sub-step is to determine the image time step corresponding to the target image feature information. The image time step can represent the position of the target image feature information in the image feature information sequence.

[0069] The second sub-step is to input the target image feature information and the historical unit fusion feature information into the convolution feature extraction unit corresponding to the image time step to generate unit fusion feature information. The historical unit fusion feature information is the unit fusion feature information output by the previous image time step corresponding to the image time step. The unit fusion feature information may be the fusion feature information of each image feature information before and including the image time step. The historical unit fusion feature information may be the fusion feature information of each image feature information before the image time step. There is a one-to-one correspondence between the image feature information in the image feature information sequence and the image time step in the time step sequence. There is a corresponding convolution feature extraction unit for each time step in the time step sequence. Each convolution feature extraction unit in the convolution feature extraction unit sequence may be connected in series. The convolution feature extraction unit may be a convolution neural network model.

[0070] In the third sub-step, in response to determining that the target image feature information is the image feature information of the target position in the image feature information sequence, the unit fusion feature information is determined as the sequence fusion feature information corresponding to the image feature information sequence. The target position may be the sequence position of the image feature information corresponding to the latest image time step in the image feature information sequence.

[0071] Sub-step 5, in response to determining that the target image feature information is not the image feature information of the target position in the image feature information sequence, determining the next time step corresponding to the image time step, wherein the next time step may be a time step in the time step sequence that is right adjacent to the image time step.

[0072] Sub-step 6: taking the image feature information corresponding to the next time step as the target image feature information, and continuing to execute the fourth generation step.

[0073] Sub-step 7: superimposing the fused feature information and the sequence fused feature information to generate superimposed feature information.

[0074] As an example, the execution subject may perform feature information superposition on the fused feature information and the sequence fused feature information in a top-down superposition manner to generate superimposed feature information.

[0075] Sub-step 8, determining the last superimposed feature information for the target divided frame image group. The last superimposed feature information may be the superimposed feature information corresponding to the last time step. The last time step may be a time step in the time step sequence that is left adjacent to the image time step.

[0076] Sub-step 9, inputting the superimposed feature information and the previous superimposed feature information into a multi-layer fused convolution unit connected in series to generate fused convolution feature information. The fused convolution unit may be a convolution model. The fused convolution feature information may be fused feature information between the superimposed feature information and the previous superimposed feature information.

[0077] Sub-step 10, inputting the fused convolution feature information into the fragment sub-description information generation layer corresponding to the target divided frame image group to generate the fragment sub-description information for the target divided frame image group. The fragment sub-description information generation layer may be a network layer for generating fragment sub-description information. The fragment sub-description information generation layer may be a multi-layer series fully connected layer.

[0078] Sub-step 11, in response to determining that the target divided frame image group is the divided frame image group at the target position in the above divided frame image group sequence, generating segment description information according to the obtained segment sub-description information sequence.

[0079] As an example, the above execution entity may directly determine the fragment sub-description information sequence as the fragment description information.

[0080] In the second step, in response to determining that the target divided frame image group is not the divided frame image group at the target position in the above-mentioned divided frame image group sequence, the next divided frame image group corresponding to the target divided frame image group is taken as the target divided frame image group, and the above-mentioned third generation step is continued.

[0081] Step 104: Add corresponding object statement information to each subtitle information in the subtitle information sequence to generate added subtitle information and obtain an added subtitle information sequence.

[0082] In some embodiments, the execution subject may add corresponding object statement information to each subtitle information in the subtitle information sequence to generate added subtitle information and obtain an added subtitle information sequence. The object statement information may be a statement representing the statement made by the object in the target video. For example, the object statement information may be "Then, object B said "this thing is not done right"".

[0083] As an example, for each subtitle information, the execution subject may use various preset words in the object statement information adding template to add corresponding object statement information to the subtitle information to generate added subtitle information.

[0084] Step 105: According to the time sequence, the obtained segment description information sequence and the above-mentioned added subtitle information sequence are correspondingly fused to generate a fused information sequence.

[0085] In some embodiments, the execution subject may fuse the obtained segment description information sequence and the added subtitle information sequence according to the time sequence to generate a fused information sequence.

[0086] As an example, the execution subject may combine the time corresponding to each segment description information and the time corresponding to the added subtitle information in order from early to late to obtain a fusion information sequence.

[0087] Step 106: Generate a video explanation text for the above fused information sequence.

[0088] In some embodiments, the execution subject may generate a video explanation text for the fusion information sequence, wherein the video explanation text may be a text explaining the video content of the target video.

[0089] In some optional implementations of some embodiments, the generating of the video explanation text for the fused information sequence may include the following steps:

[0090] The first step is to perform information encoding processing on the fused information in the above fused information sequence to generate encoded information and obtain an encoded information sequence.

[0091] As an example, the execution subject may input the fused information in the fused information sequence into a coding model to generate coding information and obtain a coding information sequence, wherein the coding model may be a word embedding model + a recurrent neural network model.

[0092] The second step is to perform the following fifth generation step for the coded information sequence:

[0093] Sub-step 1, inputting the above-mentioned coded information sequence into a discriminant model to generate discriminant information. The discriminant model can be a discriminant model in a generative and adversarial neural network model. In practice, the discriminant model can be a multi-layer convolutional neural network model. The discriminant information can be one of the following: discriminant information representing the presence of the detected coded information and discriminant information representing the absence of the detected coded information.

[0094] Sub-step 2, in response to determining that the discrimination information represents the existence of the detected coding information, determining at least one coding information subsequence corresponding to the discrimination information. The at least one coding information subsequence may be each of the detected coding information subsequences. The overall content corresponding to each of the fused information in the fused information subsequence corresponding to the coding information subsequence has a content inconsistency problem. That is, there is an information mismatch problem between each of the fused information in the obtained fused information subsequence.

[0095] Sub-step 3, for each of the at least one coding information subsequence, input the coding information subsequence into a generative model to generate a candidate coding information subsequence. The generative model may be a generative model in a generative and adversarial neural network model. The generative model may be an LSTM model. The candidate coding information subsequence may be a coding information subsequence whose corresponding time is the same as the corresponding time of the coding information subsequence.

[0096] Sub-step 4: Replace at least one coding information subsequence in the above coding information sequence with the obtained candidate coding information subsequence to obtain a replaced coding information sequence.

[0097] Sub-step 5, inputting the replaced encoded information sequence into the above-mentioned discriminant model to generate target discriminant information.

[0098] Sub-step 6, in response to determining that the target discrimination information indicates that the detected coded information does not exist, decoding the replaced coded information sequence to generate a decoded information sequence as a video explanation text.

[0099] As an example, the execution entity may input the replaced encoded information sequence into a decoding model to generate a decoded information sequence.

[0100] In the third step, in response to determining that the target discrimination information represents the existence of the detected coding information, a replaced coding information sequence corresponding to the target discrimination information is determined as the target replaced coding information sequence.

[0101] The fourth step is to use the target replaced encoded information sequence as the encoded information sequence and continue to execute the sixth generation step.

[0102] Step 107: extracting the first audio sequence corresponding to the subtitle information sequence in the target video.

[0103] In some embodiments, the execution subject may extract the first audio sequence corresponding to the subtitle information sequence in the target video, wherein the first audio in the first audio sequence and the subtitle information in the subtitle information sequence have a one-to-one correspondence.

[0104] As an example, first, the execution subject may determine a time sequence corresponding to the subtitle information sequence, and then extract a first audio sequence from the target video according to the time sequence.

[0105] Step 108: Generate a second audio sequence for the segment description information sequence.

[0106] In some embodiments, the execution subject may generate a second audio sequence for the segment description information sequence, wherein the segment description information in the segment description information sequence and the second audio in the second audio sequence have a time correspondence relationship.

[0107] As an example, the execution entity may utilize audio processing software to generate a second audio sequence for the segment description information sequence.

[0108] Step 109: Generate an explanation audio for the video explanation text according to the first audio sequence and the second audio sequence.

[0109] As an example, the above execution entity may directly combine the first audio sequence and the second audio sequence in the order of time from early to late to generate a combined audio sequence as the explanation audio.

[0110] Considering the problems of the above conventional solutions, facing the above technical problem 2: the audio content of the obtained explanation audio may not be smooth, and explanation errors may occur. Combined with the advantages / technical status, it can be decided to adopt the following solution.

[0111] In some optional implementations of some embodiments, generating an explanation audio for the video explanation text according to the first audio sequence and the second audio sequence includes the following steps:

[0112] The first step is to extract the audio features corresponding to each first audio in the above-mentioned first audio sequence as the first audio feature information, and obtain the first audio feature information sequence, and extract the audio features corresponding to each second audio in the above-mentioned second audio sequence as the second audio feature information, and obtain the second audio feature information sequence.

[0113] In the second step, according to the audio time corresponding to each first audio in the above-mentioned first audio sequence and the audio time corresponding to each second audio in the above-mentioned second audio sequence, each first audio feature information in the first audio feature information sequence and each second audio feature information in the second audio feature information sequence are correspondingly combined to generate an audio feature information sequence.

[0114] The third step is to perform the following first information generation step for the audio feature information sequence:

[0115] Sub-step 1: input the above audio feature information sequence into the discriminative neural network model in the generative and discriminative neural network models to generate discriminative information for the above audio feature information sequence.

[0116] Sub-step 2, in response to determining that the discrimination information indicates at least one audio feature information subsequence with audio semantic disfluency, for each audio feature information subsequence in the at least one audio feature information subsequence, performing the following second information generation step:

[0117] The first sub-step is to determine the time period corresponding to the above audio feature information sub-sequence as the target time period.

[0118] The second sub-step is to extend the target time period to generate an extended time period, wherein each extended time period is in a time-continuous relationship with the target time period.

[0119] The third sub-step is to determine the extended audio feature information sub-sequence corresponding to the extended time period.

[0120] The fourth sub-step is to input the above-mentioned extended audio feature information sub-sequence into the generative neural network model in the generative and discriminative neural network models to generate a candidate audio feature information sub-sequence for the target time period.

[0121] Sub-step 3: replacing at least one audio feature information subsequence in the above audio feature information sequence with at least one obtained candidate audio feature information subsequence to obtain a replaced audio feature information sequence.

[0122] Sub-step 4: input the above audio feature information sequence into a discriminative neural network model to generate discriminative information for the above audio feature information sequence as target discriminative information.

[0123] Sub-step 5, in response to determining that the target discrimination information represents an audio feature information subsequence without audio semantic disfluency, inputting the replaced audio feature information sequence into an audio generation model to generate audio as explanation audio.

[0124] In a fourth step, in response to determining that the target discrimination information represents an audio feature information subsequence that has audio semantic disfluency, at least one candidate audio feature information subsequence is determined, wherein the candidate audio feature information subsequence is an audio feature information subsequence that represents audio semantic disfluency.

[0125] The fifth step is to generate a replaced audio feature information sequence for the at least one candidate audio feature information subsequence as a target audio feature information sequence.

[0126] The sixth step is to use the target audio feature information sequence as the audio feature information sequence and continue to execute the above-mentioned first information generation step.

[0127] The content in "some optional implementations of some embodiments" as an inventive point of the present disclosure solves the technical problem mentioned in the background technology that "the audio content of the obtained explanation audio may be not smooth, and explanation errors may occur." Based on this, the present disclosure can not only effectively determine the semantic closeness between each audio feature information in the audio feature information sequence through the generative and discriminative neural network models, but also accurately adjust at least one audio feature information subsequence with low semantic closeness to generate more accurate and smoother explanation audio, thereby avoiding the occurrence of explanation errors.

[0128] Step 110, the explanation audio and the video identifier corresponding to the target video are stored in correspondence.

[0129] As an example, the execution subject may store the explanation audio and the video identifier corresponding to the target video in a corresponding manner, wherein the video identifier may represent the video identity information corresponding to the target video.

[0130] In some optional implementations of some embodiments, after step 110, the steps further include:

[0131] The first step is to parse the audio playback information for the target video in response to receiving the audio playback information for the target video to generate audio playback start node information, audio playback end node information and playback speech speed information. The audio playback information may be playback information that characterizes audio playback for the target video. In practice, the audio playback information may be the playback audio emitted by the target object. In practice, the target object may be an object with visual impairment. The audio playback start node information may be the node information of the start node of the audio playback. For example, the audio playback start node information may be the start node time of the audio playback, or the start chapter information of the audio playback. The audio playback end node information may be the node information of the end node of the audio playback. For example, the audio playback end node information may be the end node time of the audio playback, or the end chapter information of the audio playback. The playback speech speed information may be the speech speed of the audio playback explanation.

[0132] The second step is to intercept the stored explanation audio according to the audio playback start node information and the audio playback end node information to obtain the intercepted audio.

[0133] As an example, the execution subject may perform audio interception on the stored explanation audio according to the starting node time corresponding to the audio playback starting node information and the starting node time corresponding to the audio playback ending node information to obtain the intercepted audio.

[0134] The third step is to perform audio playback of the above-mentioned intercepted audio according to the speech speed corresponding to the above-mentioned playback speech speed information.

[0135] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the audio storage method of some embodiments of the present disclosure can accurately and efficiently generate the explanation audio, so that the target user can perform the audio explanation for the explanation audio later. Specifically, the reason why the relevant explanation audio is not accurate is that the explanation content of the generated explanation audio is limited and the explanation content is incoherent, resulting in poor audio explanation effect. Based on this, the audio storage method of some embodiments of the present disclosure first performs video frame extraction processing on the target video to generate a frame image sequence, so as to facilitate the subsequent extraction of the corresponding subtitle information and the generation of the corresponding text content of the explanation audio. Then, the subtitle information corresponding to the above target video is extracted to obtain the subtitle information sequence, which is used as an important text feature in the subsequent explanation audio, effectively ensuring the correctness of the key content of the subsequent explanation audio. Then, for each adjacent two subtitle information in the above subtitle information sequence, the following first generation step is performed: the first step is to determine the frame image subsequence in the above frame image sequence between the above two adjacent subtitle information, so as to facilitate the subsequent generation of segment description information without subtitle information. The second step is to generate segment description information for the above frame image subsequence to enrich the audio content of the subsequent explanation audio. Next, corresponding object statement information is added to each subtitle information in the above subtitle information sequence to generate added subtitle information and obtain an added subtitle information sequence, so that subsequent listeners can clearly understand the explanation object of the subtitle and the substantive content of the corresponding subtitle information. Further, according to the time sequence, the obtained segment description information sequence and the above added subtitle information sequence are fused with corresponding information to generate a fused information sequence to obtain a preliminary and perfect explanation text. On this basis, a video explanation text for the above fused information sequence can be accurately generated to further enhance the expressiveness and richness of the content on the basis of the fused information sequence. Secondly, the first audio sequence corresponding to the above subtitle information sequence in the above target video is extracted to obtain the corresponding audio feature content. In addition, a second audio sequence for the above segment description information sequence can be accurately generated. Furthermore, according to the above first audio sequence and the above second audio sequence, an explanation audio with not only correct but also rich content can be accurately generated for the above video explanation text. Finally, the above explanation audio and the video identifier corresponding to the above target video are stored correspondingly, so that the subsequent target user can perform audio explanation for the explanation audio.

[0136] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an audio storage device. These device embodiments are similar to Figure 1Corresponding to the method embodiments shown, the audio storage device can be specifically applied to various electronic devices.

[0137] like Figure 2 As shown, an audio storage device 200 includes: a video frame extraction processing unit 201, a first extraction unit 202, an execution unit 203, an addition unit 204, an information fusion unit 205, a first generation unit 206, a second extraction unit 207, a second generation unit 208, a third generation unit 209 and a storage unit 210. The video frame extraction processing unit 201 is configured to perform video frame extraction processing on a target video to generate a frame image sequence; the first extraction unit 202 is configured to extract subtitle information corresponding to the target video to obtain a subtitle information sequence; the execution unit 203 is configured to perform the following first generation step for each adjacent two subtitle information in the subtitle information sequence: determine a frame image subsequence in the frame image sequence between the two adjacent subtitle information; generate segment description information for the frame image subsequence; the adding unit 204 is configured to add corresponding object description information to each subtitle information in the subtitle information sequence to generate added subtitle information and obtain an added subtitle information sequence; the information fusion unit 205 is configured to generate subtitle information based on the subtitle information sequence; According to the time sequence, the obtained segment description information sequence and the above-mentioned added subtitle information sequence are fused with corresponding information to generate a fused information sequence; the first generation unit 206 is configured to generate a video explanation text for the above-mentioned fused information sequence; the second extraction unit 207 is configured to extract the first audio sequence corresponding to the above-mentioned subtitle information sequence in the above-mentioned target video; the second generation unit 208 is configured to generate a second audio sequence for the above-mentioned segment description information sequence; the third generation unit 209 is configured to generate an explanation audio for the above-mentioned video explanation text based on the above-mentioned first audio sequence and the above-mentioned second audio sequence; the storage unit 210 is configured to store the above-mentioned explanation audio and the video identifier corresponding to the above-mentioned target video in a corresponding manner.

[0138] It is understandable that the units recorded in the audio storage device 200 are similar to the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the audio storage device 200 and the units contained therein, and will not be described in detail here.

[0139] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0140] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0141] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0142] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0143] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0144] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0145] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: performs video frame extraction processing on the target video to generate a frame image sequence; extracts the subtitle information corresponding to the target video to obtain a subtitle information sequence; for each adjacent two subtitle information in the subtitle information sequence, performs the following first generation step: determines a frame image subsequence in the frame image sequence between the two adjacent subtitle information; generates segment description information for the frame image subsequence; adds corresponding object statement information to each subtitle information in the subtitle information sequence to generate added subtitle information to obtain an added subtitle information sequence; according to the time sequence, performs corresponding information fusion on the obtained segment description information sequence and the added subtitle information sequence to generate a fused information sequence; generates a video explanation text for the fused information sequence; extracts a first audio sequence corresponding to the subtitle information sequence in the target video; generates a second audio sequence for the segment description information sequence; generates an explanation audio for the video explanation text according to the first audio sequence and the second audio sequence; and stores the explanation audio and the video identifier corresponding to the target video in a corresponding manner.

[0146] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor including: a video frame extraction processing unit, a first extraction unit, an execution unit, an addition unit, an information fusion unit, a first generation unit, a second extraction unit, a second generation unit, a third generation unit and a storage unit. Among them, the names of these units do not constitute a limitation on the units themselves in certain cases. For example, the video frame extraction processing unit may also be described as "a unit that performs video frame extraction processing on a target video to generate a frame image sequence".

[0149] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0150] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. An audio storage method, comprising: Performing video frame extraction processing on the target video to generate a frame image sequence; Extracting subtitle information corresponding to the target video to obtain a subtitle information sequence; For each two adjacent subtitle information in the subtitle information sequence, the following first generation step is performed: Determine a frame image subsequence in the frame image sequence between the two adjacent subtitle information; Generating fragment description information for the frame image subsequence; Adding corresponding object statement information to each subtitle information in the subtitle information sequence to generate added subtitle information, thereby obtaining an added subtitle information sequence, wherein the object statement information is a statement wording representing the statement made by the object in the target video; According to the time sequence, the obtained segment description information sequence and the added subtitle information sequence are correspondingly fused to generate a fused information sequence; Generate a video explanation text for the fusion information sequence, wherein the video explanation text is a text that explains the video content of the target video, The generating of the video explanation text for the fused information sequence comprises: Performing information encoding processing on the fused information in the fused information sequence to generate encoding information, thereby obtaining an encoding information sequence; For the encoded information sequence, the following fifth generation step is performed: Inputting the encoded information sequence into a discriminant model to generate discriminant information; In response to determining that the discrimination information indicates the presence of the detected coded information, determining at least one coded information subsequence corresponding to the discrimination information; For each of the at least one encoding information subsequence, inputting the encoding information subsequence into a generation model to generate a candidate encoding information subsequence; Replacing at least one coding information subsequence in the coding information sequence with the obtained candidate coding information subsequence to obtain a replaced coding information sequence; Inputting the replaced encoded information sequence into the discriminant model to generate target discriminant information; In response to determining that the target discrimination information indicates that the detected coded information is absent, decoding the replaced coded information sequence to generate a decoded information sequence as a video explanation text; In response to determining that the target discrimination information represents the presence of the detected coded information, determining a replaced coded information sequence corresponding to the target discrimination information as a target replaced coded information sequence; The target replaced encoded information sequence is used as the encoded information sequence, and the sixth generation step is continued; Extracting a first audio sequence corresponding to the subtitle information sequence in the target video; generating a second audio sequence for the segment description information sequence; Generate an explanation audio for the video explanation text according to the first audio sequence and the second audio sequence, wherein: The step of generating an explanation audio for the video explanation text according to the first audio sequence and the second audio sequence comprises: Extracting audio features corresponding to each first audio in the first audio sequence as first audio feature information to obtain a first audio feature information sequence, and extracting audio features corresponding to each second audio in the second audio sequence as second audio feature information to obtain a second audio feature information sequence; According to the audio time corresponding to each first audio in the first audio sequence and the audio time corresponding to each second audio in the second audio sequence, correspondingly combining each first audio feature information in the first audio feature information sequence and each second audio feature information in the second audio feature information sequence to generate an audio feature information sequence; For the audio feature information sequence, the following first information generation step is performed: Inputting the audio feature information sequence into a discriminative neural network model in a generative and discriminative neural network model to generate discriminative information for the audio feature information sequence; In response to determining that the discriminant information represents at least one audio feature information subsequence with audio semantic disfluency, for each audio feature information subsequence in the at least one audio feature information subsequence, performing a second information generating step: Determine a time period corresponding to the audio feature information subsequence as a target time period; Extending the target time period to generate an extended time period; Determine an extended audio feature information subsequence corresponding to the extended time period; Inputting the extended audio feature information subsequence into a generative neural network model in a generative and discriminative neural network model to generate a candidate audio feature information subsequence for a target time period; Replacing at least one audio feature information subsequence in the audio feature information sequence with the obtained at least one candidate audio feature information subsequence to obtain a replaced audio feature information sequence; Inputting the audio feature information sequence into a discriminative neural network model to generate discriminative information for the audio feature information sequence as target discriminative information; In response to determining that the target discriminant information represents an audio feature information subsequence without audio semantic disfluency, inputting the replaced audio feature information sequence into an audio generation model to generate audio as explanation audio; In response to determining that the target discriminant information represents an audio feature information subsequence that has audio semantic disfluency, determining at least one candidate audio feature information subsequence, wherein the candidate audio feature information subsequence is an audio feature information subsequence that represents audio semantic disfluency; generating a replaced audio feature information sequence for the at least one candidate audio feature information subsequence as a target audio feature information sequence; Taking the target audio feature information sequence as the audio feature information sequence, and continuing to perform the first information generating step; The explanation audio and the video identifier corresponding to the target video are stored correspondingly.

2. The method according to claim 1, wherein: The method further comprises: In response to receiving the audio playback information for the target video, the audio playback information is parsed to generate audio playback start node information, audio playback end node information and playback speech speed information; According to the audio playback start node information and the audio playback end node information, the stored explanation audio is intercepted to obtain the intercepted audio; The audio playback for the intercepted audio is performed according to the speech speed corresponding to the playback speech speed information.

3. The method according to claim 1, wherein: The generating of the fragment description information for the frame image subsequence includes: Inputting each frame image in the frame image subsequence into a pre-trained image feature extraction model to generate image feature information, thereby obtaining an image feature information subsequence; Performing clustering processing on each piece of image feature information in the image feature information subsequence to generate an image feature information cluster set, wherein the time corresponding to each piece of image feature information in the image feature information cluster is continuous; According to the number of clusters, the model structure of the acquired image set partition information model is adjusted to generate an adjusted image set partition information model; Using the adjusted image set partitioning information model, generating an image feature information group set for the image feature information subsequence; Dividing the frame image subsequence according to the image feature information cluster set and the image feature information group set to generate a divided frame image group sequence; According to the divided frame image group sequence, segment description information is generated.

4. The method according to claim 3, wherein: The image set partitioning information model includes: an image set partitioning model and a plurality of image similarity determination models, wherein the number of image similarity determination models included in the plurality of image similarity determination models is a predetermined number, the predetermined number is greater than the number of clusters, and the number of image sets output by the image set partitioning model is the same as the predetermined number; and The step of adjusting the model structure of the acquired image set partition information model according to the number of clusters to generate an adjusted image set partition information model includes: According to the number of clusters, each initial partition parameter included in the image set partition model is trimmed to generate a trimmed image set partition model, wherein the number of parameters of each partition parameter included in the trimmed image set partition model is the number of clusters; According to each model output corresponding to the cropped image set partitioning model, model cropping is performed on each image similarity determination model in the multiple image similarity determination models to obtain multiple cropped image similarity determination models, wherein the input of each model in the multiple cropped image similarity determination models corresponds to each model output corresponding to the cropped image set partitioning model.

5. The method according to claim 4, wherein: The step of using the adjusted image set partitioning information model to generate an image feature information group set for the image feature information subsequence comprises: For the image feature information subsequence, the following second generation step is performed: Using the image set partitioning model, partitioning the image feature information subsequence into feature information to generate an initial image feature information group set; Inputting each initial image feature information group in the initial image feature information group set into the cropped multiple image similarity determination models to generate an image similarity set; In response to determining that the values ​​corresponding to the respective image similarities in the image similarity set are greater than a preset value, determining the initial image feature information set as the image feature information set; In response to determining that there is an image similarity whose corresponding value is less than the preset value in the image similarity set, determining at least one image similarity less than the preset value; Determine a difference between each image similarity in the at least one image similarity and the preset value to generate a difference value, thereby obtaining at least one difference value; According to the at least one phase difference value, adjusting at least one initial partitioning parameter included in the image set partitioning model to obtain an adjusted image set partitioning model; The adjusted image set partition model is used as the image set partition model, and the second generation step is continued.

6. The method according to claim 3, wherein: The step of generating segment description information according to the divided frame image group sequence comprises: For the target divided frame image group in the divided frame image group sequence, the following third generation step is performed: Determine the image feature information corresponding to each divided frame image in the target divided frame image group as the target image feature information, and obtain the target image feature information group; Inputting the target image feature information group into a feature fusion model to generate fused feature information; Sorting the feature information of each target image in the target image feature information group to generate an image feature information sequence; For the target image feature information in the image feature information sequence, the following fourth generation step is performed: Determine the image time step corresponding to the target image feature information; Inputting the target image feature information and the historical unit fusion feature information into the convolution feature extraction unit corresponding to the image time step to generate unit fusion feature information, wherein the historical unit fusion feature information is the unit fusion feature information outputted at the previous image time step corresponding to the image time step; In response to determining that the target image feature information is the image feature information of the target position in the image feature information sequence, determining the unit fusion feature information as the sequence fusion feature information corresponding to the image feature information sequence; In response to determining that the target image feature information is not the image feature information of the target position in the image feature information sequence, determining a next time step corresponding to the image time step; Taking the image feature information corresponding to the next time step as the target image feature information, and continuing to execute the fourth generation step; Superimposing the fusion feature information and the sequence fusion feature information to generate superimposed feature information; Determine the last superimposed feature information of the target divided frame image group; Inputting the superimposed feature information and the previous superimposed feature information into a multi-layer series fused convolution unit to generate fused convolution feature information; Inputting the fused convolution feature information into the fragment sub-description information generation layer corresponding to the target divided frame image group to generate the fragment sub-description information for the target divided frame image group; In response to determining that the target divided frame image group is the divided frame image group at the target position in the divided frame image group sequence, generating segment description information according to the obtained segment sub-description information sequence; In response to determining that the target divided frame image group is not the divided frame image group at the target position in the divided frame image group sequence, the next divided frame image group corresponding to the target divided frame image group is taken as the target divided frame image group, and the third generation step is continued.

7. An audio storage device, comprising: A video frame extraction processing unit is configured to perform video frame extraction processing on the target video to generate a frame image sequence; A first extraction unit is configured to extract subtitle information corresponding to the target video to obtain a subtitle information sequence; The execution unit is configured to execute the following first generation step for each two adjacent subtitle information in the subtitle information sequence: determining a frame image subsequence in the frame image sequence between the two adjacent subtitle information; Generating fragment description information for the frame image subsequence; An adding unit is configured to add corresponding object statement information to each subtitle information in the subtitle information sequence to generate added subtitle information and obtain an added subtitle information sequence, wherein the object statement information is a statement wording representing the statement made by the object in the target video; An information fusion unit is configured to fuse the obtained segment description information sequence and the added subtitle information sequence according to a time sequence relationship to generate a fused information sequence; The first generating unit is configured to generate a video explanation text for the fused information sequence, wherein the video explanation text is a text that explains the video content of the target video, and the generating of the video explanation text for the fused information sequence includes: performing information encoding processing on the fused information in the fused information sequence to generate encoding information and obtain an encoding information sequence; for the encoding information sequence, performing the following fifth generating step: inputting the encoding information sequence into a discriminant model to generate discriminant information; in response to determining that the discriminant information represents the existence of the detected encoding information, determining at least one encoding information subsequence corresponding to the discriminant information; for each encoding information subsequence in the at least one encoding information subsequence, inputting the encoding information subsequence into Generate a model to generate candidate coding information subsequences; replace at least one coding information subsequence in the coding information sequence with the obtained candidate coding information subsequence to obtain a replaced coding information sequence; input the replaced coding information sequence into the discriminant model to generate target discriminant information; in response to determining that the target discriminant information represents the absence of the detected coding information, decode the replaced coding information sequence to generate a decoded information sequence as a video explanation text; in response to determining that the target discriminant information represents the presence of the detected coding information, determine the replaced coding information sequence corresponding to the target discriminant information as the target replaced coding information sequence; use the target replaced coding information sequence as the coding information sequence and continue to perform the sixth generation step; A second extraction unit is configured to extract a first audio sequence corresponding to the subtitle information sequence in the target video; A second generating unit, configured to generate a second audio sequence for the segment description information sequence; The third generating unit is configured to generate an explanation audio for the video explanation text according to the first audio sequence and the second audio sequence, wherein the generating of the explanation audio for the video explanation text according to the first audio sequence and the second audio sequence includes: extracting the audio features corresponding to each first audio in the first audio sequence as the first audio feature information to obtain a first audio feature information sequence, and extracting the audio features corresponding to each second audio in the second audio sequence as the second audio feature information to obtain a second audio feature information sequence; extracting the audio features corresponding to each first audio in the first audio sequence as the second audio feature information to obtain a second audio feature information sequence; extracting the audio features corresponding to each second audio in the second audio sequence according to the audio time corresponding to each first audio in the first audio sequence and the audio time corresponding to each second audio in the second audio sequence time, correspondingly combining each first audio feature information in the first audio feature information sequence and each second audio feature information in the second audio feature information sequence to generate an audio feature information sequence; for the audio feature information sequence, performing the following first information generating step: inputting the audio feature information sequence into the discriminant neural network model in the generative and discriminant neural network models to generate discriminant information for the audio feature information sequence; in response to determining that the discriminant information represents at least one audio feature information subsequence with audio semantic disfluency, for each audio feature information subsequence in the at least one audio feature information subsequence, performing the second information generating step: determining whether the audio feature information subsequence is fluent The method comprises the steps of: selecting a time period corresponding to the target time period as the target time period; extending the target time period to generate an extended time period; determining an extended audio feature information subsequence corresponding to the extended time period; inputting the extended audio feature information subsequence into a generative neural network model in a generative and discriminative neural network model to generate a candidate audio feature information subsequence for the target time period; replacing at least one audio feature information subsequence in the audio feature information sequence with at least one obtained candidate audio feature information subsequence to obtain a replaced audio feature information sequence; inputting the audio feature information sequence into a discriminative neural network model to generate discriminant information for the audio feature information sequence as the target discriminant information; In response to determining that the target discriminant information represents an audio feature information subsequence without audio semantic disfluency, input the replaced audio feature information sequence into an audio generation model to generate audio as explanation audio; in response to determining that the target discriminant information represents an audio feature information subsequence with audio semantic disfluency, determine at least one candidate audio feature information subsequence, wherein the candidate audio feature information subsequence is an audio feature information subsequence representing the presence of audio semantic disfluency; generate a replaced audio feature information sequence for the at least one candidate audio feature information subsequence as a target audio feature information sequence; and continue to perform the first information generation step using the target audio feature information sequence as the audio feature information sequence; The storage unit is configured to store the explanation audio and the video identifier corresponding to the target video in correspondence.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images