Method, apparatus, device and storage medium for information processing

Abstract

Embodiments of the present disclosure relate to a method, apparatus, device and storage medium for information processing. The method proposed herein comprises: processing a video content with a motion encoder to determine motion features, and processing the video content with an image encoder to determine image features; constructing input features for a model based on the motion features, the image features and audio features of an audio content; and providing the input features to the model to determine a matching degree of the video content and the audio content. Thus, embodiments of the present disclosure can evaluate the matching degree of the video content and the audio content from the images and motions of the video content and the audio content, thereby improving the accuracy of the matching degree of the video content and the audio content.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatus, devices and computer-readable storage media for information processing. Background Technology

[0002] In recent years, with the rapid development of video technology, more and more video works have been presented in people's lives. These video works are widely produced in various fields such as video production, game development, film post-production, and multimedia interactive systems. To improve the effect of video works, they usually include various appropriate sound effects. The precise matching of sound effects and visuals can significantly improve the quality of the work and the audience's immersion. Therefore, how to more accurately evaluate the matching degree between audio and video has become a focus of attention. Summary of the Invention

[0003] In a first aspect of this disclosure, an information processing method is provided. The method includes: processing video content using a motion encoder to determine motion features, and processing the video content using an image encoder to determine image features; constructing input features for a model based on the motion features, image features, and audio features of the audio content; and providing the input features to the model to determine the degree of matching between the video content and the audio content.

[0004] In a second aspect of this disclosure, an apparatus for information processing is provided. The apparatus includes: a first determining module configured to process video content using a motion encoder to determine motion features, and to process the video content using an image encoder to determine image features; a constructing module configured to construct input features for a model based on the motion features, image features, and audio features of the audio content; and a second determining module configured to provide the input features to the model to determine the degree of matching between the video content and the audio content.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0010] Figure 2 A flowchart illustrating an example evaluation process for the degree of matching between audio and video content according to some embodiments of this disclosure is shown;

[0011] Figure 3 A flowchart illustrating an example process of information processing according to some embodiments of this disclosure is shown;

[0012] Figure 4 A schematic structural block diagram of an example apparatus for information processing according to some embodiments of the present disclosure is shown; and

[0013] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0018] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0019] As mentioned above, precise matching of sound effects and visuals can significantly enhance the quality of a work and the audience's immersion. Traditional techniques assess the degree of matching between sound effects and visuals by extracting features from each separately. However, this assessment method can only evaluate the semantic matching degree between sound effects and visuals globally, and cannot assess whether the sound effects and visuals match in detail. Therefore, this assessment method is difficult to accurately evaluate the degree of matching between sound effects and visuals.

[0020] The embodiments of this disclosure propose an information processing scheme. According to the scheme, a motion encoder can be used to process video content to determine motion features, and an image encoder can be used to process video content to determine image features; further, input features of the model can be constructed based on motion features, image features, and audio features of audio content; additionally, input features can be provided to the model to determine the degree of matching between video content and audio content.

[0021] Based on this approach, the embodiments of this disclosure can comprehensively consider the degree of matching between the motion features and image features of the video content and the audio features, thereby enabling the obtained degree of matching between the video content and the audio content to include not only the overall matching degree of the audio-visual content, but also the matching degree of the audio-visual rhythm, thus enabling the evaluation of the accuracy of the matching degree between the video content and the audio content.

[0022] Therefore, the embodiments of this disclosure can evaluate the matching degree of video content and audio content from multiple perspectives, including images and actions, thereby improving the accuracy of the matching degree of video content and audio content.

[0023] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0024] Example environment

[0025] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In this example environment 100, an electronic device 110 can support the invocation of a model 120, which can be deployed in the electronic device 110 or in other electronic devices and supported by the electronic device 110. The electronic device 110 can utilize such a model 120 to process audio content 131 and video content 132, thereby obtaining a matching degree 140 between the audio content 131 and the video content 132.

[0026] Electronic device 110 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, server devices, etc. Terminal devices may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the terminal device may also support any type of user-facing interface (such as "wearable" circuitry).

[0027] Server equipment can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server equipment may include, for example, computing systems / servers such as mainframes, edge computing nodes, and computing devices in cloud environments. Server equipment can support electronic device 110 calling model 120.

[0028] A communication connection can be established between the server device and the terminal device. This communication connection can be established via wired or wireless means. The communication connection may include, but is not limited to, Bluetooth, mobile network, Universal Serial Bus (USB), and Wireless Fidelity (WiFi) connections, and the embodiments of this disclosure are not limited in this respect. In the embodiments of this disclosure, the server device and the terminal device can achieve signaling interaction through the communication connection between them.

[0029] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0030] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0031] Example process of assessing a degree of match

[0032] The following will be referenced Figure 2 This document describes some example processes for evaluating the degree of matching between audio content and video content according to embodiments of the present disclosure. Figure 2 The diagram illustrates an example evaluation process for the degree of matching between audio and video content according to some embodiments of the present disclosure. Process 200 may, for example, be performed by... Figure 1 The electronic device 110 shown is provided.

[0033] In some embodiments, the electronic device 110 can acquire audio and video content to be matched. For example, such audio and video content to be matched can have the same duration, thereby being encoded into corresponding features with the same number of channels, thus improving matching accuracy. For example, such audio and video content to be matched can be uploaded by a user, or stored locally by the electronic device 110, or stored in the cloud. For example, the electronic device 110 can batch evaluate the matching degree of multiple audio-video pairs (audio-video pairs are also called combinations of audio content and video content). The embodiments of this disclosure take evaluating the matching degree between one audio content and one video content as an example; for specific implementations of batch evaluation, please refer to the embodiments of this disclosure.

[0034] The following is based on Figure 2 The process 200 shown, taking the matching of audio content 210 and video content 220 by electronic device 110 as an example, provides an exemplary description of the scheme for evaluating the matching degree of audio content and video content provided by the embodiments of this disclosure.

[0035] In some embodiments, the electronic device 110 can determine audio features 261 (also known as audio representations) of such audio content 210. As an example, such audio features 261 can be achieved using appropriate audio feature determination tools, such as neural network models capable of determining audio features 261. As an example, the electronic device 110 can input such audio content 210 into an attention mechanism neural network model to obtain such audio features 261.

[0036] To improve the accuracy of determining such audio features 261, the electronic device 110 may first determine the Mel spectrum of the audio content 210 and encode such Mel spectrum to obtain the audio features 261 of the audio content 210. As an example, the electronic device 110 may first utilize a Mel spectrum determination tool to determine the Mel spectrum of the audio content 210 (e.g., a tensor of dimension [T1, D1]). Further, the electronic device 110 may input such Mel spectrum of the audio content 210 into an audio encoder (e.g., a one-dimensional convolutional neural network 230) to obtain the audio features 261 of the audio content 210. It should be understood that the electronic device 110 may also utilize other audio feature 261 determination tools to determine the audio features 261 of the audio content 210, and the embodiments of this disclosure are not intended to limit the specific implementation of such audio feature determination.

[0037] In some embodiments, the audio feature 261 may have a length inconsistent with the video content 220, which could lead to low accuracy in assessing the matching degree between the audio and video content. Therefore, the electronic device 110 can add downsampling neural network layers, pooling layers, and other network layers to the audio feature determination tool to adjust the dimensionality of the audio feature 261. For example, based on the added downsampling neural network layers, pooling layers, and other network layers, the electronic device 110 can obtain an audio feature 261 with reduced dimensionality. For example, the dimension of the audio feature 261 can be [T2, D2], where T1 can be a multiple of T2 (e.g., 4 times, 8 times, etc.).

[0038] Furthermore, the electronic device 110 can determine the video content features of such video content 220 to evaluate the matching degree between audio content 210 and video content 220. As an example, the electronic device 110 can determine the video tensor corresponding to the video content 220 in a specified dimension (e.g., dimension [8,224,224,3], etc., such specified dimension can be adjusted according to actual conditions) to improve the adaptability of the embodiments of this disclosure to various common video content sources.

[0039] As an example, electronic device 110 can adjust the video attributes of video content 220 (e.g., at least one of resolution and frame rate) to obtain a video tensor corresponding to such video content 220 that satisfies preset video attributes. For example, such preset video attributes can be specified dimensions, which can indicate the number of frames, width, height, and / or number of channels of the video content. For example, specifying the dimension [8, 224, 224, 3] can indicate video content with a frame rate of 8, a resolution of [224, 224], a height of 224, and 3 channels. Based on such specified dimensions, the information processing scheme provided by the embodiments of this disclosure can have higher compatibility.

[0040] Furthermore, the electronic device 110 can determine the motion features 262 (also known as motion representations) and image features 263 (also known as image representations) corresponding to such video tensors, respectively. Thus, the electronic device 110 can construct the input features of model 270 based on such audio features 261, motion features 262, and image features 263, and then use model 270 to determine the degree of matching between audio content 210 and video content 220. As an example, such motion features 262 and image features 263 can contain different types of information in the video content. For example, motion features can include information corresponding to transient sound effects in the video content, and image features can include image information of each video frame in the video content. As an example, such model 270 can include any suitable model (e.g., model 140), which can, for example, output a classification result 280 indicating the degree of matching between the corresponding audio content and video content based on the received fused features 260.

[0041] As an example, such motion features 262 and image features 263 can be obtained using different video feature determination tools. As an example, such motion features 262 can be determined using a motion encoder (e.g., a 3D convolutional neural network 240). Specifically, the electronic device 110 can use a motion encoder to compare the image information of each video frame of the video content 220 with the image information of its corresponding adjacent video frames, thereby determining the image motion information corresponding to each video frame. As an example, the adjacent video frame corresponding to one of the video frames of the video content 220 can be a video frame preceding that video frame and / or a video frame following that video frame. Further, the electronic device 110 can use a motion encoder to encode the image motion information corresponding to each video frame to determine motion features 262. Thus, the electronic device 110 can capture information corresponding to transient sound effects in the video content 220.

[0042] As an example, such image feature 263 can be determined by processing with image encoder 250. Specifically, electronic device 110 can process each video frame of video content 220 using image encoder 250 to determine the image information of each video frame, and then determine the image features of video content 220 by encoding the image information of each video frame. As an example, electronic device 110 can input the video tensor of video content 220 into image encoder 250, which can be configured to encode the input video tensor, thereby obtaining such image feature 263 containing global image information of each video frame. As an example, such image encoder 250 can be any suitable image encoder, and this disclosure is not intended to limit the specific structure of such image encoder.

[0043] It should be understood that such action features 262 and image features 263 can also be implemented using other video feature determination models, such as pre-trained Transformer models, etc. The embodiments disclosed herein are not intended to limit the specific video feature determination tools used for video feature determination.

[0044] In some embodiments, the electronic device 110 can determine the degree of matching between audio content 210 and video content 220 by calculating the vector distance between such audio features 261 and such motion features 262 (and / or image features 263). Such matching degree evaluation schemes have low accuracy; therefore, in embodiments of this disclosure, the electronic device 110 fuses such audio features 261, motion features 262, and image features 263 to obtain the input features of the model (e.g., fused feature 260). As an example, the electronic device 110 can fuse such audio features 261, motion features 262, and image features 263 at the channel dimension. For example, the audio features 261, motion features 262, and image features 263 per second can be concatenated (or otherwise fused) to form the fused feature 260 corresponding to that second.

[0045] Furthermore, the electronic device 110 can utilize model 270 to classify such fused features 260 to obtain a classification result 280. As an example, such a classification result 280 can indicate the degree of matching between audio content 210 and video content 220. As an example, the electronic device 110 can determine such a classification result 280 from multiple candidate classifications corresponding to different degrees of matching. Thus, the electronic device 110 can determine the degree of matching between video content 220 and audio content 210 based on such a classification result 280.

[0046] In some embodiments, the electronic device 110 can adjust the scope of the classification result 280 based on the sound effect type of the audio content 210. For example, if the sound effect type of the audio content 210 is ambient sound effect (e.g., the audio content 210 is audio containing ambient sound effects), the electronic device 110 can determine the classification result 280 from a first candidate category and a second candidate category. If the sound effect type of the audio content is transient sound effect (e.g., the audio content 210 is audio containing transient sound effects), the electronic device 110 can determine the classification result 280 from a first candidate category, a second candidate category, and a third candidate category. As an example, the different matching degrees corresponding to the first candidate category, the second candidate category, and the third candidate category indicate the matching degree between the audio content 210 and the video content 220 at least in terms of semantics and rhythm.

[0047] As an example, the first matching degree corresponding to the first candidate category (e.g., labeled 2, or other suitable identifier) ​​indicates that the video semantics of the video content and the audio semantics of the audio content match, and the video rhythm of the video content and the audio rhythm of the audio content match; the second matching degree corresponding to the second candidate category (e.g., labeled 0, or other suitable identifier) ​​indicates that the video semantics of the video content and the audio semantics of the audio content do not match, and the video rhythm of the video content and the audio rhythm of the audio content do not match; the third matching degree corresponding to the third candidate category (e.g., labeled 1, or other suitable identifier) ​​indicates that the video semantics of the video content and the audio semantics of the audio content match, but the video rhythm of the video content and the audio rhythm of the audio content do not match. Therefore, the electronic device 110 can further improve the accuracy of the classification result 280, and thus further improve the accuracy of evaluating the matching degree between the audio content 210 and the video content 220.

[0048] To improve the accuracy of such classification result 280, electronic device 110 can pre-train such model 270 using a sample set. As an example, such a sample set may include a reference sample video and a set of extended sample videos generated based on the reference sample video. This set of extended sample videos can be generated by performing sample augmentation on the reference sample video, such as video content cropping, video content flipping, video content rotation, video content brightness adjustment, and frame dropping. Specifically, such a set of extended sample videos can be generated based on at least one of the following sample augmentation processes: cropping at least a portion of the reference sample video; flipping at least a portion of the reference sample video; rotating at least a portion of the reference sample video; adjusting the brightness of the reference sample video; and deleting at least a portion of the reference sample video frames. Therefore, embodiments of this disclosure can improve the generalization ability of model 270 for various scenarios, thereby improving the accuracy of classification result 280.

[0049] In some embodiments, such a model 270 may further include a model with a rotational position coding layer, which the electronic device 110 can then utilize to process the input features to determine the degree of matching between the audio content 210 and the video content 220. Based on such a rotational position coding layer, model 270 can process input features of various lengths, thereby improving the generalization ability of such model 270 for various audio and video content to be matched.

[0050] In some embodiments, the electronic device 110 can also perform content filtering on such audio content 210 to improve the accuracy of such audio features, thereby improving the accuracy of such classification results 280. As an example, the electronic device 210 can remove audio content segments containing music, semantics, or other audio content from the audio content 210 based on the audio content itself, thereby increasing the proportion of sound effects content in the audio content 210.

[0051] In some embodiments, such an information processing scheme can be used to construct training data for an audio generation model, thereby improving the quality of the trained audio generation model. Specifically, the electronic device 110 can evaluate the matching degree between multiple pairs of audio and video content using the aforementioned scheme for evaluating the matching degree between audio and video content. Furthermore, the electronic device 110 can use audio and video content with matching degrees satisfying a preset matching degree to construct training data, and train the audio generation model based on such training data. As an example, the electronic device 110 can add such training data to a training dataset and use the training dataset to further improve the quality of the audio generation model by analyzing audio and video content. Based on this, the electronic device 110 can obtain an audio generation model with higher audio generation quality, and thus, the electronic device 110 can use such an audio generation model to generate audio that better matches the video.

[0052] Based on this approach, the embodiments of this disclosure can comprehensively consider the degree of matching between the motion features and image features of the video content and the audio features, thereby enabling the obtained degree of matching between the video content and the audio content to include not only the overall matching degree of the audio-visual content, but also the matching degree of the audio-visual rhythm, thus enabling the evaluation of the accuracy of the matching degree between the video content and the audio content.

[0053] Therefore, the embodiments of this disclosure can evaluate the matching degree of video content and audio content from multiple perspectives, including images and actions, thereby improving the accuracy of the matching degree of video content and audio content.

[0054] Example process

[0055] Figure 3 A flowchart of an example process 300 for information processing according to some embodiments of the present disclosure is shown. Process 300 can be implemented at electronic device 110. Reference is made below. Figure 1 Describe the process 300.

[0056] like Figure 3As shown, at box 310, electronic device 110 uses a motion encoder to process video content to determine motion features and an image encoder to process video content to determine image features.

[0057] At box 320, electronic device 110 constructs the input features of the model based on action features, image features, and audio features of audio content.

[0058] At box 330, electronic device 110 provides input features to the model to determine the degree of matching between video content and audio content.

[0059] In some embodiments, processing video content using a motion encoder to determine motion features includes: using a motion encoder to compare the image information of each video frame of the video content with the image information of the corresponding adjacent video frames to determine the image motion information corresponding to each video frame; and determining motion features based on the image motion information corresponding to each video frame.

[0060] In some embodiments, processing video content using an image encoder to determine image features includes: processing each video frame of the video content using an image encoder to determine image information for each video frame; and determining image features by encoding the image information for each video frame.

[0061] In some embodiments, the audio characteristics of the audio content are determined based on the following process: determining the Mel spectrum of the audio content; and encoding the Mel spectrum using an audio encoder to determine the audio characteristics.

[0062] In some embodiments, action features, image features, and audio features correspond to the same multiple feature channels, and the input features of the model are constructed based on action features, image features, and audio features by fusing action features, image features, and audio features based on multiple feature channels to construct the input features of the model.

[0063] In some embodiments, before processing the video content using a motion encoder, process 300 further includes: adjusting the video properties of the video content to satisfy preset video properties, wherein the video properties include at least one of resolution and frame rate.

[0064] In some embodiments, providing input features to the model to determine the degree of matching between video content and audio content includes: providing input features to the model to determine a classification result for the input features, wherein the classification result is determined from a plurality of candidate classifications corresponding to different degrees of matching; and determining the degree of matching between the video content and audio content based on the classification result.

[0065] In some embodiments, determining the classification result includes: determining the classification result from a first candidate category and a second candidate category in response to the audio content's sound effect type being an ambient sound effect type; or determining the classification result from a first candidate category, a second candidate category, and a third candidate category in response to the audio content's sound effect type being a transient sound effect type.

[0066] In some embodiments, the first matching degree corresponding to the first candidate category indicates that the video semantics of the video content and the audio semantics of the audio content match, and the video rhythm of the video content and the audio rhythm of the audio content match; the second matching degree corresponding to the second candidate category indicates that the video semantics of the video content and the audio semantics of the audio content do not match, and the video rhythm of the video content and the audio rhythm of the audio content do not match; the third matching degree corresponding to the third candidate category indicates that the video semantics of the video content and the audio semantics of the audio content match, but the video rhythm of the video content and the audio rhythm of the audio content do not match.

[0067] In some embodiments, the model utilizes a rotational position encoding layer to process input features to determine the degree of matching; the rotational position encoding layer is configured to encode input features of varying lengths.

[0068] In some embodiments, the model is trained on a sample set, which includes a reference sample video and a set of extended sample videos generated based on the reference sample video.

[0069] In some embodiments, a set of extended sample videos is generated based on at least one of the following sample enhancement processes: cropping at least a portion of a reference sample video; flipping at least a portion of a reference sample video; rotating at least a portion of a reference sample video; adjusting the brightness of a reference sample video; and deleting at least a portion of video frames from a reference sample video.

[0070] In some embodiments, process 300 further includes: in response to the matching degree of video content and audio content satisfying a preset matching degree, constructing training data based on video content and audio content to add training data to a training dataset; and training an audio generation model using the training dataset, wherein the audio generation model is configured to generate audio that matches the video.

[0071] Example apparatus and devices

[0072] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an example apparatus 400 for information processing according to certain embodiments of the present disclosure is shown. Apparatus 400 may be implemented as or included in electronic device 110. Various modules / components in apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0073] like Figure 4 As shown, the device 400 includes a first determining module 410, configured to process video content using a motion encoder to determine motion features and to process video content using an image encoder to determine image features; a building module 420, configured to build input features of a model based on motion features, image features, and audio features of audio content; and a second determining module 430, configured to provide input features to the model to determine the degree of matching between video content and audio content.

[0074] In some embodiments, the first determining module 410 is further configured to: use an action encoder to compare the image information of each video frame of the video content with the image information of the corresponding adjacent video frames to determine the image action information corresponding to each video frame; and determine action features based on the image action information corresponding to each video frame.

[0075] In some embodiments, the first determining module 410 is further configured to: process each video frame of the video content using an image encoder to determine the image information of each video frame; and determine image features by encoding the image information of each video frame.

[0076] In some embodiments, the apparatus 400 further includes a third determining module configured to: determine the Mel spectrum of the audio content; and encode the Mel spectrum using an audio encoder to determine audio features.

[0077] In some embodiments, action features, image features, and audio features correspond to the same multiple feature channels, and the construction module 420 is further configured to: fuse action features, image features, and audio features based on multiple feature channels to construct the input features of the model.

[0078] In some embodiments, the apparatus 400 further includes an adjustment module configured to: adjust the video attributes of the video content to satisfy preset video attributes, wherein the video attributes include at least one of resolution and frame rate.

[0079] In some embodiments, the second determining module 430 is further configured to: provide input features to the model to determine a classification result for the input features, wherein the classification result is determined from a plurality of candidate classifications corresponding to different degrees of matching; and determine the degree of matching between the video content and the audio content based on the classification result.

[0080] In some embodiments, the second determining module 430 is further configured to: determine a classification result from a first candidate category and a second candidate category in response to the audio content's sound effect type being an ambient sound effect type; or determine a classification result from a first candidate category, a second candidate category, and a third candidate category in response to the audio content's sound effect type being a transient sound effect type.

[0081] In some embodiments, the first matching degree corresponding to the first candidate category indicates that the video semantics of the video content and the audio semantics of the audio content match, and the video rhythm of the video content and the audio rhythm of the audio content match; the second matching degree corresponding to the second candidate category indicates that the video semantics of the video content and the audio semantics of the audio content do not match, and the video rhythm of the video content and the audio rhythm of the audio content do not match; the third matching degree corresponding to the third candidate category indicates that the video semantics of the video content and the audio semantics of the audio content match, but the video rhythm of the video content and the audio rhythm of the audio content do not match.

[0082] In some embodiments, the model utilizes a rotational position encoding layer to process input features to determine the degree of matching; the rotational position encoding layer is configured to encode input features of varying lengths.

[0083] In some embodiments, the model is trained on a sample set, which includes a reference sample video and a set of extended sample videos generated based on the reference sample video.

[0084] In some embodiments, a set of extended sample videos is generated based on at least one of the following sample enhancement processes: cropping at least a portion of a reference sample video; flipping at least a portion of a reference sample video; rotating at least a portion of a reference sample video; adjusting the brightness of a reference sample video; and deleting at least a portion of video frames from a reference sample video.

[0085] In some embodiments, the apparatus 400 further includes a training module configured to, in response to a preset matching degree between the video content and the audio content, construct training data based on the video content and the audio content to add the training data to the training dataset; and train an audio generation model using the training dataset, wherein the audio generation model is configured to generate audio that matches the video.

[0086] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0087] Figure 5A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 Electronic devices 110 or Figure 4 Device 400.

[0088] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0089] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0090] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0091] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0092] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0093] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0094] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0095] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0096] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0098] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An information processing method, comprising: A motion encoder is used to process the video content to determine motion features, and an image encoder is used to process the video content to determine image features; Based on the action features, the image features, and the audio features of the audio content, the input features of the model are constructed; as well as The input features are provided to the model to determine the degree of matching between the video content and the audio content.

2. The method of claim 1, wherein processing the video content using the motion encoder to determine the motion features comprises: Using the motion encoder, the image information of each video frame of the video content is compared with the image information of its corresponding adjacent video frames to determine the image motion information corresponding to each video frame; and The motion features are determined based on the image motion information corresponding to each video frame.

3. The method of claim 1, wherein processing the video content using the image encoder to determine the image features comprises: The image encoder is used to process each video frame of the video content to determine the image information of each video frame; as well as The image features are determined by encoding the image information of each video frame.

4. The method of claim 1, wherein the audio features of the audio content are determined based on the following process: Determine the Mel spectrum of the audio content; and The Mel spectrum is encoded using an audio encoder to determine the audio characteristics.

5. The method according to claim 1, wherein the action features, the image features, and the audio features correspond to the same plurality of feature channels, and the input features of the model constructed based on the action features, the image features, and the audio features include: Based on the multiple feature channels, the action features, image features, and audio features are fused to construct the input features of the model.

6. The method of claim 1, wherein before processing the video content using a motion encoder, the method further comprises: Adjust the video attributes of the video content to meet preset video attributes, wherein the video attributes include at least one of resolution and frame rate.

7. The method of claim 1, wherein providing the input features to the model to determine the degree of matching between the video content and the audio content comprises: The input features are provided to the model to determine a classification result for the input features, wherein the classification result is determined from multiple candidate classifications corresponding to different degrees of matching; as well as Based on the classification results, the degree of matching between the video content and the audio content is determined.

8. The method of claim 7, wherein determining the classification result comprises: In response to the audio content being classified as an ambient sound effect, the classification result is determined from a first candidate category and a second candidate category. or In response to the audio content being classified as a transient sound effect type, the classification result is determined from a first candidate category, a second candidate category, and a third candidate category.

9. The method according to claim 8, wherein: The first matching degree corresponding to the first candidate category indicates that the video semantics of the video content and the audio semantics of the audio content match, and that the video rhythm of the video content and the audio rhythm of the audio content match. The second matching degree corresponding to the second candidate category indicates that the video semantics of the video content and the audio semantics of the audio content do not match, and that the video rhythm of the video content and the audio rhythm of the audio content do not match. The third matching degree corresponding to the third candidate classification indicates that the video semantics of the video content and the audio semantics of the audio content match, while the video rhythm of the video content and the audio rhythm of the audio content do not match.

10. The method of claim 1, wherein the model utilizes a rotational position encoding layer to process the input features to determine the degree of matching, the rotational position encoding layer being configured to encode input features of different lengths.

11. The method of claim 1, wherein the model is trained based on a sample set, the sample set including reference sample videos and a set of extended sample videos generated based on the reference sample videos.

12. The method of claim 11, wherein the set of extended sample videos is generated based on at least one of the following sample augmentation processes: At least a portion of the reference sample video is cropped; Flip at least a portion of the reference sample video; Rotate at least a portion of the reference sample video; Adjust the brightness of the reference sample video; Delete at least a portion of the video frames from the reference sample video.

13. The method according to claim 1, further comprising: In response to the matching degree between the video content and the audio content meeting a preset matching degree, training data is constructed based on the video content and the audio content, and the training data is added to the training dataset; as well as An audio generation model is trained using the training dataset, wherein the audio generation model is configured to generate audio that matches the video.

14. An apparatus for information processing, comprising: The first determining module is configured to process video content using a motion encoder to determine motion features, and to process the video content using an image encoder to determine image features; The construction module is configured to construct the input features of the model based on the action features, the image features, and the audio features of the audio content; as well as The second determining module is configured to provide the input features to the model to determine the degree of matching between the video content and the audio content.

15. An electronic device comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images